Grafana: The open and composable observability platform
Blog post from Grafana Labs
High cardinality in a Prometheus environment refers to the situation where a database has a vast number of unique time series, often resulting from metrics with quickly changing label values. This can lead to increased resource usage, higher costs, and degraded performance of both the database and the applications generating the metrics. High cardinality can negatively impact applications by consuming excessive CPU and memory resources, slowing down operations, and causing errors in queries and Grafana dashboards due to data limits. While labels are essential for telemetry, they should be used judiciously to avoid these issues, and alternative observability pillars like logs and traces can provide necessary information without increasing cardinality. Developers are encouraged to review and optimize their metric labels to prevent high cardinality and collaborate with their Prometheus management teams to address any existing issues. Tools like Grafana Cloud's Adaptive Metrics can help manage high cardinality by aggregating series to alleviate the load on the system.