May 2022 Summaries

On-call rotations, essential for handling system incidents, can significantly impact engineers' well-being, as highlighted in the exploration of strategies used by Honeycomb to monitor on-call health. The traditional approach of counting disruptions, such as alarms and incidents, is critiqued for its emphasis on negative outcomes and lack of consideration for the subjective experiences of engineers. Instead, Honeycomb adopts a more nuanced method inspired by Erik Hollnagel's resilience grid, focusing on abilities to respond, monitor, learn, and anticipate. This approach includes regular surveys using a simplified Google Form to capture qualitative feedback on these dimensions, though it acknowledges limitations in capturing the full complexity of on-call experiences, such as rotation size, alert volume, and organizational trust. The data, while not perfect, aids in informing operational decisions and is shared anonymously within the organization to improve planning and interventions. Honeycomb continues to refine its methodology to ensure the feedback remains relevant and useful, promoting a more supportive on-call environment.

Amazon has introduced C7g instances powered by the AWS Graviton3 processors, which promise significant performance improvements over their predecessors. Honeycomb, having successfully migrated its entire Amazon EC2 fleet to Arm64-based AWS Graviton family instances, is enthusiastic about leveraging Graviton3 for enhanced performance and scalability. The company experienced notable benefits from its migration to Graviton2, including lower costs, increased RAM, and reduced latency, and expects even greater advantages from Graviton3. With seamless integration across AWS services and infrastructure managed through tools like Terraform and Chef, Honeycomb has been able to optimize workloads and reduce complexity. Testing has shown a 30% performance improvement from Graviton2 to Graviton3, and Honeycomb's infrastructure now operates almost exclusively on Arm processors, benefiting from increased speed and efficiency. This transition aligns with the company's goal of offering cutting-edge observability tools while maintaining a low total cost of ownership, allowing Honeycomb to focus on innovation rather than operational overhead.

The OpenTelemetry Collector is a flexible service that can be deployed in various ways within an infrastructure, either as a sidecar next to applications or centrally as a network service, to manage telemetry data efficiently. It offers significant benefits such as centralizing configurations, which is advantageous for managing API keys and other secrets, and centralizing egress, which is particularly useful in secure networks where direct internet connections are restricted. Additionally, the Collector can filter and secure trace data by redacting sensitive information through processors and improve application performance by managing synchronous trace data sending, thus reducing latency impacts. It also enriches telemetry data with additional metadata like cloud regions or Kubernetes namespace information. The decision to use the Collector depends on individual needs and preferences, as it involves a trade-off between adding more management layers and achieving centralized control, with the option to seek further guidance from developer advocates if necessary.

Developers often face challenges identifying the root causes of 500 errors reported by large customers, which can lead to unresolved issues. Using the observability platform Honeycomb, developers can efficiently pinpoint the origins of these errors by analyzing Application Load Balancer (ALB) logs. The guide explains how to integrate ALB logs with Honeycomb using a step-by-step AWS CloudFormation setup, enabling data to flow seamlessly for quicker issue identification. By leveraging Honeycomb's query engine, users can generate Heatmaps and utilize the BubbleUp feature to visualize and isolate problematic status codes, such as 500 errors, and explore data dimensions to uncover patterns and potential causes like authentication issues. This method allows teams to better assess the impact on specific customers and create derived columns for quicker insights, ultimately improving incident response times and customer communication. Honeycomb's free tier offers an accessible entry point for developers to explore its capabilities without the pressure of sales engagement.

Modern observability tools like Honeycomb differ from older distributed tracing systems such as Dapper by offering greater ease of adoption and more comprehensive data collection. While older systems were designed for hyper-scale environments and relied on sampling a very small percentage of requests, which made it difficult to capture the full scope of performance issues, modern tools collect a weighted proportional amount of raw trace data. This approach allows users to easily analyze latency distributions and visualize data in different formats, making debugging more efficient and flexible. Additionally, modern observability tools do not require tracing the entire system to be effective; users can gain insights by instrumenting just a single service, with the value of the tool increasing as more services are connected, allowing for detailed filtering, searching, and visualization of trace data.

"Observability Engineering: Achieving Production Excellence" has finally been completed after three years of writing, with the authors expressing relief and gratitude for the time it took, given the significant evolution in the field of observability during that period. Initially expected to be completed quickly, the project was delayed as the authors navigated a changing landscape, marked by the rise of OpenTelemetry as a standard and a shift in understanding technical concepts such as high cardinality, high dimensionality, and wide structured events. Over these years, early adopters have grown into experts, and innovations like visualizing CI/CD pipelines as traces have emerged. The authors also adapted their database to accommodate serverless architecture, allowing for cost-effective management of numerous production workloads. Now available for pre-order on Amazon, the book is also offered as a free download from Honeycomb. Additionally, the authors are launching "The Authors’ Cut," a series exploring the more opinionated content left out of the book, featuring discussions and live demos on key observability concepts.

Observability Engineering, authored by Honeycomb's Charity Majors, Liz Fong-Jones, and George Miranda, and published by O’Reilly Media, offers a comprehensive guide on implementing observability to enhance business velocity and software performance. This book provides a practical framework for adopting observability, focusing on structured events, hypothesis verification, and analysis loops, tailored to modern cloud-native, microservice architectures. Observability tools are essential for overcoming the limitations of traditional monitoring in distributed systems, enabling engineering teams to swiftly diagnose and resolve application issues, thereby aligning technical efforts with business goals. The publication addresses the increasing demand for observability solutions amid the shift from monolithic to distributed systems, positioning itself as a vendor-neutral, objective resource amidst a competitive and often confusing marketplace. Honeycomb emphasizes that understanding observability is crucial for improving customer experiences and accelerating release cycles, as evidenced by its adoption by companies like HelloFresh, Stripe, and Slack.