Handling missing metrics in Wavefront
Blog post from Box
Box relies heavily on real-time metrics to manage its large-scale Kubernetes clusters, using alerts to notify deviations from expected behavior. Two significant types of deviations are identified: anomalies in metric values and the cessation of data generation. A major challenge encountered was that many alerts were not correctly formulated to account for missing data, which led to critical analysis and adjustments in alert formulation. This includes considering the delay time for notifications to balance prompt reactions and minimize false positives. Different monitoring systems like Influxdata, Prometheus, DataDog, and Wavefront handle missing data differently, with Wavefront offering specific solutions like the mcount function and the NO_DATA state. Box found success using the default() function to address missing data, using a strategy that fills gaps in metrics with a default value after a specified delay. This approach helps maintain robust alerts and prevents alert fatigue by minimizing false positives from temporary data glitches.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.