July 2024 Summaries

In examining a significant database migration incident at Honeycomb that caused a major outage, the author discusses the construction of errors and their subsequent impact on corrective measures. The incident was triggered by an ENUM modification on a database table that unexpectedly led to a full table rewrite, highlighting the risks of database migrations and the potential for blame within engineering processes. Despite the structured process involving validation and verification checks, errors still occurred, pointing to a need for improved detection of risky migrations. The author suggests that linting MySQL migrations could provide a low-cost feedback loop to catch risky patterns, balancing between training all engineers to be SQL experts and relying on post-issue repair work. Though the newly implemented linter has yet to identify significant issues, it offers a proactive approach to mitigating migration risks by enforcing algorithm specifications and allowing engineers to override checks when necessary. The piece underscores the importance of ownership and experimentation in process improvement, especially during on-call shifts, as a means to address latent system issues and enhance reliability.

The concept of a Center of Production Excellence (CoPE) aims to introduce change within organizations by disrupting the status quo to encourage exploration and innovation, as modeled by Laurent Hébert-Dufresne's research on social organization and behavioral norms. The CoPE seeks to implement coordinated changes that promote prosocial behaviors over individualistic achievements, with a focus on both bottom-up and top-down practices that mutually reinforce each other. These practices include socio-technical reforms at various organizational levels and emphasize the importance of observability using tools like Honeycomb. Observability is achieved through the instrumentation of software to produce detailed telemetry data, enabling developers to analyze system behavior through high-cardinality, wide events. OpenTelemetry (OTel) is highlighted as a tool for auto-instrumentation, facilitating the transition from traditional metrics to distributed tracing, ultimately aiding developers in focusing on customer experience and production excellence. The introduction of structural constraints, such as requiring Honeycomb queries in pull requests, helps embed these practices into the workflow, promoting long-term cooperation that benefits the group.

In the second part of the blog post detailing a chaos engineering experiment, the team tested their infrastructure's resilience by simulating the failure of an entire availability zone (AZ) using AWS’s Fault Injection Service (FIS). By abruptly terminating EC2 instances, blocking network communication, and preventing new instance creation, they aimed to understand the impact on their systems. The experiment revealed a specific issue with AWS's PrivateLink, which routed some requests to the affected AZ, resulting in HTTP 504 errors. The team preemptively mitigated this by instructing the Application Load Balancer (ALB) to avoid using the compromised AZ. Despite initial concerns from telemetry data, the actual impact was minimal, with minor HTTP 500 errors affecting about one percent of requests for 31 seconds, and their infrastructure continued to handle the load effectively. Kubernetes rapidly launched new instances to replace those terminated, and within hours, their infrastructure returned to normal. The test confirmed the system's resilience and provided insights for future improvements, encouraging similar experiments to ensure service reliability.

Pair programming is emphasized as a vital practice in software development, particularly in remote-first environments, due to its ability to foster collaboration and shared understanding, enhancing both personal and codebase development. The narrative reflects on the author's experiences with pair programming at a non-profit coding bootcamp, Techtonica, and later as an intern and junior software engineer at Honeycomb, highlighting the importance of effective communication and feedback in these sessions. The author shares insights into best practices for successful pair programming, including the importance of honoring time, setting clear goals, and providing constructive feedback, while also acknowledging the mutual benefits for junior and senior developers alike, as juniors gain technical skills and seniors hone leadership abilities. The piece underscores that successful pair programming creates a symbiotic relationship that enhances individual growth and contributes positively to a company's culture and productivity.

Honeycomb conducted a bold chaos engineering experiment by intentionally disrupting one-third of its production infrastructure using AWS’s Fault Injection Service to test system resilience. This approach, which involved both non-production and production environments, aimed to uncover unexpected failure modes and improve system reliability without impacting customer service. Initial tests revealed issues like a Zookeeper lock acquisition bug, false telemetry alerts, and AWS PrivateLink traffic disruptions, which were addressed through bug fixes and mitigation strategies. By simulating failures in key components like the Shepherd service and observing a coincidental test of Kafka, Honeycomb gained confidence in its infrastructure's ability to handle an Availability Zone (AZ) failure. The company decided to test during peak traffic hours to ensure readiness and minimize risks associated with off-peak incident responses, demonstrating a commitment to learning from real-world scenarios to enhance reliability.

Establishing a Center of Production Excellence (CoPE) requires recruiting the right change agents who can adapt to the organization's needs and promote resilience, emphasizing both operational excellence and customer experience. A key challenge for CoPE members is proactively convincing the organization to change before issues arise, as they work within complex systems subject to constraints. Strategies for recruiting effective CoPE members include enlisting respected engineers who advocate for change and leveraging data from tools like Honeycomb to identify potential candidates. Diversity in team composition is crucial, as it brings varied backgrounds and skills that enhance problem-solving capabilities. Once recruited, members should regularly meet to develop a charter outlining objectives, problems to address, and strategies for change, ensuring alignment with organizational goals. Obtaining commitment from organizational stakeholders for this charter is essential for empowering the CoPE to initiate effective change.

As a launch customer of Amazon EC2 R8g instances powered by AWS Graviton4, this blog shares detailed performance insights and challenges encountered during the transition from Graviton3 to Graviton4. The Graviton4 processors demonstrated significant improvements, including a 22-24% decrease in processing time for various requests and a 10% reduction in CPU utilization while handling 23% more requests per task. However, issues arose with network buffering and slow-sending clients, necessitating a reevaluation of instrumentation to accurately measure network and processing times. Additionally, the Graviton4 processors dropped support for the 32-bit instruction state, causing compatibility issues with certain software. Despite these challenges, the transition to Graviton4 has resulted in more than double the throughput per vCPU compared to the start of their Graviton journey, underscoring the advantages of leveraging AWS's silicon innovation without the need for significant capital expenditure. The migration to Graviton4 is expected to enhance Honeycomb's ability to provide best-in-class observability at a predictable price, supporting distributed tracing and trace-level analytics across numerous microservices.

Generative AI is experiencing rapid growth, drawing parallels to the crypto bubble, but its true value lies in augmentation rather than full automation. Phillip Carter, in his Monitorama 2024 talk, emphasizes the importance of using generative AI for specific, well-defined tasks rather than attempting to automate complex human activities. He highlights the necessity of instrumentation and data evaluation in improving AI models, pointing out that the intersection of machine learning and reliability engineering provides valuable insights. Carter's experience with Honeycomb's Query Assistant underscores the significance of refining data and context to enhance AI performance, advocating for a proactive approach to addressing the shortcomings of generative AI. He encourages skeptics to contribute to the development of AI systems by focusing on observability and reliability, demonstrating that thoughtful integration and continuous iteration can lead to more effective AI tools.