June 2018 Summaries

The text explores the nuanced definitions and distinctions between events and structured logs within the context of observability, often referred to as "o11y." While all events can be represented as structured logs, not all structured logs qualify as events. Logs are collections of messages, sometimes structured with formats like JSON, conveying information about system operations. An event, however, encapsulates detailed information about what it takes for a service to perform a unit of work, including input data, computed attributes, service conditions, and outcomes. The text illustrates these concepts with examples from HTTP transactions and web server access logs, emphasizing that events are conceptual abstractions, whereas structured logs are one representation of those abstractions. It concludes by encouraging further discussion to better define these terms and suggests using Honeycomb for visualizing events.

Honeycomb's recent blog post delves into their internal use of tracing to enhance their system's performance and reliability. Tracing has been integrated as part of their ongoing dogfooding efforts to optimize their bespoke distributed columnar data store, which consists of a high-volume write path and a more complex read path. The post highlights how tracing serves both as a debugging tool and an understanding tool, allowing non-experts to grasp the control flow and performance bottlenecks in their query handling processes. By employing tracing, Honeycomb has moved beyond relying on folklore and hand-written comments to gain insights into live production systems, effectively identifying areas for improvement and validating code changes. The company has found that thoughtful instrumentation of the code, beyond just using automatic middleware, is crucial for meaningful insights, and emphasizes the importance of trace discovery and aggregation in forming and testing hypotheses about system performance. Honeycomb continues to refine their tracing capabilities, inviting feedback from the community while showcasing their commitment to advancing systems observability.

Honeycomb Tracing is a new tool designed to enhance understanding and visualization of request execution in distributed systems by providing both individual trace visualization and the capability to analyze trace data for patterns and historical trends. It uses waterfall diagrams to illustrate execution history, helping identify code structure issues, inefficiencies, and missing instrumentation. The tool aggregates data at read time, allowing users to seamlessly switch between time-series graphs, traces, and raw data without needing to predict important metrics ahead of time. Honeycomb supports OpenTracing instrumentation and is compatible with Zipkin libraries, but it also allows for trace reconstruction from structured logs or events and offers Honeycomb Beelines for easy single-service tracing with certain programming languages. Honeycomb Tracing aims to improve system performance and robustness and is now available with all Honeycomb plans, with further details and a live demo webinar scheduled for those interested in exploring its capabilities.