August 2020 Summaries

Honeycomb recently experienced a series of five incidents between July 28th and August 6th, impacting their production and dogfooding environments, with three incidents leading to partial customer-facing outages. These incidents were triggered by a mix of provider issues, such as a DNS outage in Amazon's Route53, and internal changes, including problematic code deployments that affected the query engine and storage engine database migrations. Despite the quick recovery efforts of their on-call team and effective Service Level Objectives (SLOs) that aided in early detection, the cluster of incidents prompted Honeycomb to conduct a comprehensive meta-incident review to identify common factors. They discovered that the recent incidents often involved configuration code at the seams between different system layers, areas where automated testing is challenging, and institutional knowledge gaps due to team changes. Honeycomb plans to enhance their automated checks to better simulate customer traffic, investigate improvements in their deployment process for faster rollback capabilities, and invest in additional documentation and training to address these challenges. They aim to complete these improvements by the end of Q3 to ensure higher reliability and availability for their customers.

The tutorial provides a comprehensive guide for developers looking to integrate OpenTelemetry into their Go applications and send instrumentation data to Honeycomb, starting with creating a free Honeycomb account. It walks users through setting up automatic instrumentation for HTTP and gRPC requests by using request routers and handlers, adding custom context through OpenTelemetry Span Attributes, and ensuring the configuration of data export to Honeycomb. The guide also emphasizes the importance of context propagation for distributed tracing across multiple services, offering detailed instructions for implementing OpenTelemetry hooks in HTTP and gRPC client calls. Once the application is instrumented and deployed, developers can utilize Honeycomb's UI to analyze and query their data for insights, with the support of the Honeycomb Pollinators Slack community for any questions or further discussions on observability practices.

On April 16, an unnoticed incident occurred in which approximately 10% of event traffic to API hosts was rejected due to four new API servers responding with 401 and 500 errors for about 1.5 hours, caused by a database migration error. This issue was discovered a week later when a customer reported missing data. The investigation revealed that the error stemmed from an automated deployment that proceeded despite an incomplete database migration, leading new hosts to request a non-existent database column, resulting in 500 errors and subsequent 401 errors due to null API keys. The incident underscored the importance of thorough observability and data retention, which allowed the team to diagnose the problem retrospectively and implement changes to prevent similar occurrences. The incident also highlighted the need for more comprehensive SLO monitoring, including 401 errors, to better capture potential disruptions in user experience.

The session "All Aboard! Bring Your Team Together" aimed at empowering internal champions of Honeycomb and Observability, guiding them on how to engage their teams and management in adopting observability practices through Honeycomb's tools. Key strategies include enabling Single Sign-On (SSO) for seamless user onboarding, integrating Slack for sharing insights during incidents, and setting up automated alerts for proactive incident management. The session also emphasized the use of markers to provide external system context and boards for organizing investigative queries, while encouraging the addition of field descriptions to enhance team understanding of data. For further guidance on fostering a culture of observability, participants were directed to download an e-guide titled "Developing a Culture of Observability."