Generating metrics from traces with cardinality control: A closer look at HyperLogLog in Tempo

Tempo, an open-source distributed tracing backend, has introduced a powerful feature in its 2.10 release that addresses the challenge of generating metrics directly from traces, specifically focusing on managing cardinality. Through the optional metrics-generator component, Tempo can produce RED metrics from traces and push them to Mimir, filling the gap for applications that use tracing without metrics. However, this capability risks causing a cardinality explosion—an overwhelming number of unique metric series that can increase costs. To tackle this, Tempo employs HyperLogLog, a probabilistic data structure that efficiently estimates cardinality with minimal memory usage. This approach allows for significant memory savings, providing estimates with only a 3% error margin. Tempo's integration with Grafana Cloud further enhances cardinality management by enabling users to visualize, manage, and forecast metric usage, aiding in decision-making around quota adjustments and cost control.