March 2018 Summaries

Observability, as distinct from traditional monitoring, is the capability to ask new questions of a system without modifying code or gathering new data, focusing on unknown-unknowns rather than known-unknowns. It involves understanding system internals by querying external data, which is crucial in managing complex systems. True observability requires access to raw, unaggregated data, support for high-cardinality dimensions, structured logs, and incorporating observability directly into the development process. It empowers engineers to take ownership of their systems by providing timely, relevant information needed for informed decision-making. Observability is about mastering and comprehending the systems engineers build, ensuring they can make informed decisions based on real data rather than assumptions.

Honeycomb is in the early stages of transitioning some services to Kubernetes (k8s) and highlights the importance of observability in managing clusters, which can quickly become complex with numerous deployments, pods, and services. Tools like kops facilitate the initial setup of Kubernetes, but ongoing monitoring is crucial, as demonstrated by a recent issue Honeycomb faced with unexpectedly high node CPU utilization in their cluster. The company used their platform to diagnose the problem, discovering that their Honeycomb Kubernetes Agent, running as a DaemonSet, was misconfigured to consume logs from all pods, leading to increased CPU consumption in the kube-system namespace. By adjusting the configuration and using a Honeycomb Trigger to monitor CPU usage per node, they effectively addressed the situation and ensured scalability and efficiency in cluster management. Honeycomb encourages users to explore their Kubernetes integrations to gain insights into cluster events and metrics.