Detect, act, keep the evidence: node reliability across three clouds
Blog post from ClickHouse
Operating ClickHouse Cloud across thousands of nodes on AWS, GCP, and Azure exposed a gap in Kubernetes and provider-specific monitoring: machines could remain marked Ready despite disk, kernel, or network failures that disrupted workloads. To improve consistency, ClickHouse built a cross-cloud system using Node Problem Detector (NPD) to turn logs and active checks into persistent Kubernetes conditions, Node Problem Mitigator (NPM) to evaluate, drain, and when configured terminate unhealthy nodes, and Node Debug Collector (NDC) to gather diagnostic data before or after a node becomes unreachable. Mitigations can be tuned from observation to termination and are bounded by safeguards such as dry-run mode, exclusions, a kill switch, and rate limits; diagnostic collection is best-effort so it does not delay removing a faulty node. Since rollout, the system has remediated unhealthy nodes, improved visibility into failure patterns, and made evidence available for investigation and cloud-provider support, while adding complexity and risks such as false positives or excessive node removal. The system does not yet act on external cloud-provider health signals or fully unresponsive nodes.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Kubernetes | 15 | 3,885 | 364 | 106 | -25% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.