Our journey taking Kubernetes state metrics to the next level

The Datadog Containers team contributed to the kube-state-metrics project, a popular open-source Kubernetes service that generates metrics about the state of objects in a Kubernetes cluster. The team was facing challenges scaling the tool to their needs, including high data volumes and performance issues. To address these challenges, they designed an extensible solution that utilized the Datadog Cluster Check feature. This solution allowed them to reduce network latency, memory footprint, and CPU usage, while also improving scalability and extensibility. The team's contribution introduced a new Kubernetes State Metrics check in the Datadog Agent, which runs a long-running thread that pulls data from the kube-state-metrics process. This resulted in significant performance improvements, including reduced execution time and memory footprint. The new solution enables users to monitor any custom resource and generate metrics for CRDs, providing a seamless experience for internal Datadog users and contributing back to the upstream code base.