January 2022 Summaries

Implementing observability in pre-production environments, such as local, development, performance test, and staging, is crucial for identifying and debugging issues early, ensuring that traces, spans, and attributes in code provide valuable insights. Observability allows developers to detect schema conflicts and refine the production schema for clarity and reliability, offering advantages over traditional logging, test outputs, and graphs by providing more powerful distributed tracing and query-time aggregation capabilities. Originating from the need to manage complex microservices and cloud systems, observability remains the most effective tool for speeding up the debugging process and enhancing both the tools and skills of developers, even in scenarios where relying on logs and outputs might seem sufficient.

Booking.com relies on a vast infrastructure of over 50,000 servers to handle more than 1.5 million room nights booked daily, necessitating robust reliability measures. To address this, Booking.com has developed two key tools: the Reliability Collaboration Model (RCM) and the Ownership Map. The RCM outlines the responsibilities for maintaining system reliability, categorizing tasks into four main areas: Basic Operations, Disaster Recovery, Observability, and Advanced Operations. It employs three Support Levels to manage the division of labor between reliability and product teams, ranging from full autonomy for product teams at Level 3 to complete support from reliability teams at Level 1. The Ownership Map visually represents system support levels and criticality, helping align reliability efforts with business priorities. This approach fosters collaboration across teams, mitigating the ownership confusion that arises when multiple product teams work on shared systems. However, system criticality is determined by product leadership, ensuring alignment with business needs. While the RCM and Ownership Map are vital tools, Booking.com acknowledges the long-term effort required to instill a reliability culture across the organization, including ongoing education, process refinement, and strategic alignment.

When dealing with long-lived streaming RPC workloads, setting service-level objectives (SLOs) can be challenging due to the absence of a clear "success" metric per stream and the potential for streams to last several days. The suggested approach involves instrumenting the workload to provide regular health updates by creating a root span for each stream per minute, known as a "tick," with a span duration of 60 seconds. This setup allows tracking of successful versus failed writes and delays against Kafka offsets, creating metric-like data that can be used to feed SLOs. By aggregating data through a stream ID and using minute-long spans, it becomes possible to monitor each stream's behavior without accumulating excessive spans or waiting for the stream to conclude. This method enables setting SLOs on the number of successful or failed connections per minute and even on individual message success rates, thus ensuring continuous observability and management of the streaming workload.

Honeycomb Developer Week featured a collaborative event where experts from companies like Apollo GraphQL, Cloudflare, LaunchDarkly, and PagerDuty shared insights on enhancing observability skills. Highlights from Day 3 included new Honeycomb features such as CONCURRENCY for visualizing running spans, RATE operators for rate of change visualization, and auto query on zoom for time-range analysis. Reliable alerting was demonstrated through integration with PagerDuty, which enhances Honeycomb Triggers for quick and efficient incident response. Dawn Parzych's session on feature flags emphasized their role in progressive delivery, allowing developers to test new features with low-risk users before broader deployment. Additional sessions covered Honeycomb’s integration with CircleCI for efficient build visualization, Apollo GraphQL for deeper request tracing, and Cloudflare Workers for tracing requests in serverless architectures. The event underscored the importance of building observability into development workflows to improve system performance and reliability.

Introducing observability without cluttering the codebase can be achieved through OpenTelemetry's auto-instrumentation, which eliminates the need for manually writing tracing code. Traditionally, tracing required explicitly creating spans and managing trace context, leading to code clutter and obscured logic. However, OpenTelemetry offers a solution where standard web frameworks can automatically handle instrumentation, allowing developers to focus on adding meaningful data to traces without excessive manual intervention. This open-source project, backed by numerous contributors and companies like Honeycomb, supports various programming languages by using techniques like global variables and runtime hooks to integrate tracing seamlessly. With auto-instrumentation, code clarity is enhanced, as tracing becomes a background process, thus allowing developers to gain insights into network requests, database calls, and more while maintaining a clean codebase. OpenTelemetry's expansive registry supports multiple languages and frameworks, ensuring that telemetry remains consistent across different observability products.

The blog post explores the use of instrumentation, specifically through OpenTelemetry, to gain insights into system performance by generating telemetry data, even when working with code written by others. It highlights the benefits of auto-instrumentation, which automatically adds trace spans by attaching hooks into popular tools and frameworks, and explains how this can be combined with manual instrumentation to enrich the data collected. The post provides an example of using auto-instrumentation with a Java application and emphasizes the importance of capturing additional attributes to understand specific application logic and user interactions. It also discusses the role of OpenTelemetry as a vendor-neutral project for providing instrumentation across various programming languages, and the ease of sending telemetry data to platforms like Honeycomb for visualization and analysis. The article concludes by encouraging readers to explore OpenTelemetry and Honeycomb's resources to enhance observability and debug issues more effectively.

Bringing observability to databases involves creatively extracting and analyzing data from these systems, which are often considered "black boxes" in software management. Tools like Honeycomb can facilitate this process by allowing users to monitor query performance, such as execution times and index usage, through detailed logging and analysis. While true observability, characterized by direct instrumentation and rapid feedback loops, is typically reserved for software that a company writes and ships, it is still possible to glean significant insights from managed software like databases by maximizing the use of logs, internal statistics, and command line tools. This approach, although more akin to monitoring, enables the correlation of database behavior with application performance, providing valuable information for performance optimization even when direct instrumentation isn't feasible.

High-performance engineering teams thrive by integrating empathy and observability into their practices, as highlighted in a webinar featuring Charity Majors, CTO of Honeycomb, and James Governor, Co-founder of RedMonk. Empathy is crucial, particularly in managing on-call responsibilities, where service level objectives (SLOs) can reduce unnecessary alerts, preventing burnout and fostering a sense of pride among team members. A healthy organizational culture encourages experimentation, allowing for systems that can fail without affecting customers, thus promoting learning and testing. Insights from industry leaders, such as those from Slack, Lacework, Heroku, HelloFresh, Vanguard, and Honeycomb, emphasize the importance of understanding telemetry data, lowering barriers to observability, and prioritizing developer happiness over sheer productivity. Observability is championed as a key tool not only for improving production but also for enhancing development processes, with the ultimate goal of creating optimal user experiences.

In 2021, understanding and operating production systems is more complex than in 2001 due to the shift from monolithic architectures to distributed microservice ecosystems hosted on cloud infrastructure. This transition, driven by the need for scalability and flexibility, has resulted in systems that are categorized as complex or even chaotic, according to the Cynefin framework, which differentiates between obvious, complicated, complex, and chaotic systems based on their predictability and cause-effect clarity. Monolithic systems, typically falling into the complicated category, allowed experts to use intuition and predefined analysis methods to diagnose issues. In contrast, modern complex systems require a "probe-sense-respond" approach to uncover "unknown unknowns" through exploratory and iterative probing, leading to the rise of observability tools that facilitate deep inspection and rapid decision-making based on real-time data. This shift acknowledges that traditional debugging methods are insufficient for today's intricate systems and underscores the importance of using advanced observability techniques to navigate the complexities inherent in modern software architecture.