Resolving High CPU Usage in Kubernetes With Honeycomb

Honeycomb is in the early stages of transitioning some services to Kubernetes (k8s) and highlights the importance of observability in managing clusters, which can quickly become complex with numerous deployments, pods, and services. Tools like kops facilitate the initial setup of Kubernetes, but ongoing monitoring is crucial, as demonstrated by a recent issue Honeycomb faced with unexpectedly high node CPU utilization in their cluster. The company used their platform to diagnose the problem, discovering that their Honeycomb Kubernetes Agent, running as a DaemonSet, was misconfigured to consume logs from all pods, leading to increased CPU consumption in the kube-system namespace. By adjusting the configuration and using a Honeycomb Trigger to monitor CPU usage per node, they effectively addressed the situation and ensured scalability and efficiency in cluster management. Honeycomb encourages users to explore their Kubernetes integrations to gain insights into cluster events and metrics.