Is Kubernetes Monitoring Flawed?

Kubernetes monitoring faces challenges due to the overwhelming volume of metrics that are often unnecessary, with a default setup potentially resulting in tens of thousands of metrics, many of which are unused in dashboards, alerts, or records. In a discussion on the OpenObservability Talks podcast with Aliaksandr Valialkin, CTO at VictoriaMetrics, the issues of high churn rate of pod metrics, proliferation of low-usage metrics, and configuration complexity were explored. Aliaksandr suggests that Kubernetes developers should focus on identifying and standardizing essential metrics to manage and reduce this data overload, as relying on default collections is not practical for most users. The conversation also emphasized the need for community-driven guidance to establish best practices and standards for metric collection, which could help users navigate the growing complexity of Kubernetes monitoring. Additionally, projects like VictoriaMetrics offer solutions to some of these challenges by providing efficient and user-friendly monitoring tools.