Content Deep Dive

Detect, act, keep the evidence: node reliability across three clouds

Blog post from ClickHouse

Post Details
Company
Date Published
Author
-
Word Count
3,151
Company Posts That Month
28
Language
English
Hacker News Points
-
Post removed?
No
Summary

Operating ClickHouse Cloud across thousands of nodes on AWS, GCP, and Azure exposed a gap in Kubernetes and provider-specific monitoring: machines could remain marked Ready despite disk, kernel, or network failures that disrupted workloads. To improve consistency, ClickHouse built a cross-cloud system using Node Problem Detector (NPD) to turn logs and active checks into persistent Kubernetes conditions, Node Problem Mitigator (NPM) to evaluate, drain, and when configured terminate unhealthy nodes, and Node Debug Collector (NDC) to gather diagnostic data before or after a node becomes unreachable. Mitigations can be tuned from observation to termination and are bounded by safeguards such as dry-run mode, exclusions, a kill switch, and rate limits; diagnostic collection is best-effort so it does not delay removing a faulty node. Since rollout, the system has remediated unhealthy nodes, improved visibility into failure patterns, and made evidence available for investigation and cloud-provider support, while adding complexity and risks such as false positives or excessive node removal. The system does not yet act on external cloud-provider health signals or fully unresponsive nodes.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Kubernetes 15 3,885 364 106 -25%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.