February 2022 Summaries

To measure the time a user waits between pressing a button and seeing the result across page loads, store the start time in sessionStorage when the button is pressed and retrieve it on the new page to start and end a trace span. This approach addresses the loss of span data caused by page loads, as the span is created on the new page with the start time explicitly set from the stored value. This method is effective for creating a single span that crosses pages, but for continuing an existing trace, additional context must also be stored in sessionStorage.

Once critical of metrics, the text's author reflects on their importance and limitations, especially in the context of observability. Metrics are recognized as a mature, essential technology for monitoring system health and third-party software, valuable for understanding hardware interactions, and gauging aggregate statistics. However, metrics are not suited for observability, which demands event-driven insights to understand software's impact on user experience and system behavior. At Honeycomb, metrics are integrated to complement observability, enhancing the understanding of code changes on systems. The company's focus remains on observability, acknowledging metrics' role in the broader ecosystem but emphasizing innovation in observability as a relatively new field. Honeycomb Metrics serves as a bridge, helping users transition from traditional monitoring to a more comprehensive observability approach.

OpenTelemetry Context is a complex abstraction that helps manage the flow of spans in distributed tracing by acting as a "you are here" marker, indicating where each span fits within a trace. Despite its simplicity as a key-value store, the context’s special feature is its ability to follow the thread of execution across distributed systems, encoding information into network calls to maintain a cohesive trace. This mechanism varies by programming language, with different implementations such as ThreadLocal storage in Java and AsyncLocalStorage in Node.js. The context is immutable, necessitating the creation of a new context when spans need to be linked as parent spans, particularly when integrating with spans created by libraries. This process involves extracting the active context, creating a new span, and establishing a new context, allowing for seamless integration of custom spans with auto-instrumented spans. Such distributed tracing provides a powerful tool for understanding software causality, although mastering it can be challenging, prompting the offer of further guidance from experts.

After a year at Honeycomb as the first site reliability engineer (SRE), the author reflects on their role in shaping the SRE function, which was initially undefined. Tasked with creating a charter to guide the SRE responsibilities, the author outlines key areas such as owning the reliability roadmap, leading incident practices, and providing tools to improve system reliability. This charter is not only a communication tool but also helps align priorities and facilitate team growth as more SREs are hired. The author uses a monthly grid to track their time allocation across different SRE tasks, allowing for rebalancing when necessary. The emphasis is on adaptability, as the SRE role evolves with the company's growth, organizational changes, and emerging challenges. Engaging with the wider SRE community, the author invites others to share their experiences to foster a collaborative learning environment.

Before the advent of continuous integration (CI), software release cycles were plagued by complexity and manual processes, often resulting in slow and error-prone deployments. The introduction of CI/CD pipelines revolutionized this by automating the build and test processes, enabling faster and more reliable deployments. However, even with CI/CD, understanding what happens in production remains crucial for quickly identifying and resolving errors. In a collaborative effort, CircleCI's Ryan Pederson and Honeycomb's Liz Fong-Jones demonstrate how integrating observability into CI/CD pipelines using Honeycomb can offer real-time insights into build health and performance. By utilizing Honeycomb's buildevents orb with CircleCI, developers can visualize build events as traces, allowing for efficient debugging and optimization of build workflows. Through Honeycomb's interface, users can compare build properties over time to identify anomalies, such as increases in tarball size, and gain a comprehensive understanding of build performance. The integration of CircleCI with Honeycomb empowers teams to gain visibility into every step of their builds, leading to more informed decisions and smoother deployments, eliminating the uncertainty often associated with build failures.

OpenTelemetry is a comprehensive standard for application instrumentation designed with intricate abstractions by experts, which can pose challenges for developers unfamiliar with telemetry. It primarily involves using a Tracer and a TracerProvider, where the TracerProvider is responsible for sending telemetry data and can provide a Tracer to construct spans, essential for enabling observability in applications. The process varies across programming languages, requiring developers to import the OpenTelemetry API and request a Tracer, with language-specific syntax for starting spans. While OpenTelemetry API packages offer no-op Tracers by default—ensuring that applications without full OpenTelemetry setups remain functional—real telemetry data transmission requires implementing a TracerProvider configured with necessary components like Exporters and SpanProcessors. Setting up a global TracerProvider is crucial for effective telemetry, and examples in different languages, such as JavaScript and Go, are provided to help in configuring this setup.

Demo days at Honeycomb are a weekly tradition where the entire company gathers on Zoom for internal project demonstrations, fostering a sense of progress and excitement about new feature developments. Originating in the R&D department, these sessions are open to all employees, providing a platform to celebrate accomplishments, enhance visibility, and involve colleagues not directly engaged in product development. Key guidelines include no demo shaming, avoiding slide decks unless relevant, and keeping demonstrations short, live, and engaging, ideally not exceeding five minutes. The tradition has become a vibrant part of the company's culture, with a dedicated Slack channel for sharing recordings and humorous GIFs, adding to the celebratory atmosphere. The idea of demo days was inspired by the author's experience at Chef Software, where it was highly successful, unlike a less frequent, monthly attempt at a previous company. The concise, rapid-fire format at Honeycomb encourages incremental progress sharing, making the sessions enjoyable and dynamic, akin to watching a story unfold in a TV series.

Observability and business intelligence (BI) tools serve distinct purposes, with observability focusing on real-time, exploratory insights into software behavior in production, and BI tools emphasizing structured, repeatable analysis of historical data. Observability tools prioritize speed and recentness, often using dynamic sampling to provide rapid, albeit approximate, insights crucial for understanding software performance and user interactions as they occur. In contrast, BI tools and data warehouses are engineered for accuracy and long-term data retention, using predefined schemas to process and analyze large volumes of data, thus enabling strategic decision-making over extended periods. Monitoring tools complement these by efficiently aggregating and aging out detailed data, striking a balance between the ephemerality of observability data and the comprehensive, structured nature of BI data. Overall, observability is essential for immediate feedback and debugging in software development, while BI supports broader business strategy through in-depth data analysis.

Honeycomb enhances team collaboration and debugging by offering comprehensive observability into both systems and user activity within the platform. Users can access a shared query history to view and learn from teammates' past queries, which are stored indefinitely for quick access without needing to re-run them. This feature is particularly beneficial during incident management, allowing team members to quickly assist in investigations and review debugging timelines. Users can name and save useful queries, making them easily accessible for future reference and enabling insight sharing among team members. Honeycomb encourages collaborative learning and improvement through its community Slack channel, inviting users to share feedback and ideas for enhancing the platform's collaborative experience.

Honeycomb was awarded the Best Observability Solution for 2021 by TechStrong Group at the DevOps Dozen² Awards, highlighting the growing importance of observability tools in modern application development. This award, announced at the Predict 2022 conference, recognizes Honeycomb's ability to provide high-performance engineering teams with the tools needed to quickly visualize, analyze, and enhance cloud application performance. Unlike traditional APM monitoring tools that rely on siloed data types, Honeycomb's observability platform offers exploratory capabilities, enabling teams to analyze vast amounts of data and swiftly address issues, thereby improving uptime, user experience, and business outcomes. The platform is trusted by major companies like HelloFresh, Stripe, and Slack for its ability to enhance incident response times, optimize performance, and accelerate release cycles, underscoring the shift from traditional APM tools to more comprehensive observability solutions that cater to complex, distributed systems.