August 2017 Summaries

Golang has become a popular language for various applications, and at Honeycomb, the new pprof web UI has been utilized to enhance observability in Golang programs. The article discusses using the new pprof UI to address a memory usage issue in the honeyelb program, which integrates with AWS Elastic Load Balancer to analyze access logs. The pprof tool, installed alongside graphviz, allows for interactive profiling in a web browser, helping identify memory-intensive functions such as libhoney.NewEvent and runtime methods related to map usage. By using the UI to refine the analysis, it was discovered that excessive Publisher structs were being spun up, leading to unnecessary memory consumption, which was then rectified. The piece emphasizes the importance of an observability mindset, using tools like pprof and Honeycomb to proactively address and prepare for system issues, reducing unknowns and enhancing system understanding.

In a detailed exploration of their engineering practices, Honeycomb discusses the significance of distinguishing between correlation and causation, particularly in the context of software operations and feature deployment. The company highlights a case where feature flags were pivotal in identifying misleading correlations during the rollout of a storage format change. Initially, Honeycomb's deployment of a new "varstring" feature seemed to correlate with slower write speeds, sparking concerns that were later mitigated by understanding the role of a hidden factor – the volume of data writes. This scenario exemplifies Simpson's Paradox, where the apparent slowdowns were not due to the varstring feature itself but rather the shift in dataset composition, emphasizing the importance of critical analysis of data and not relying solely on surface-level metrics. Honeycomb's approach showcases the value of using feature flags and observability tools to gain precise insights into system performance and maintain confidence in progressive delivery strategies.