July 2021 Summaries

In the exploration of developing Honeycomb's Service Level Objectives (SLO) product, the company adopted an unconventional approach by initially using existing tools that weren't expected to scale, yet they surprisingly did for a significant period. The development process involved leveraging the high-performance query engine "Retriever" to manage expensive queries, which allowed Honeycomb to gather valuable user feedback and iterate on the product. Despite initial doubts about scalability, the system served effectively for over a year, allowing the team to learn and adapt before implementing a more robust, scalable solution with the new stream processor "Beagle." Honeycomb's experience underscores the importance of focusing on immediate product needs and iterative development rather than prematurely optimizing for scale, ultimately leading to a successful and scalable SLO system.

Rich Anakor, chief solutions architect at Vanguard, spearheaded an initiative to enhance customer experience by improving the understanding of Vanguard's complex production environment across its engineering teams, despite the challenges posed by transitioning workloads from on-prem data centers to the public cloud. During his 2021 hnycon keynote, Rich explained how Vanguard's existing application performance monitoring approach was inadequate for their evolving infrastructure, leading his team to adopt OpenTelemetry and Honeycomb for better observability. Starting with one service, they gradually expanded their efforts, proving the effectiveness of this new approach by significantly reducing the time required to identify service and data dependencies, which facilitated cultural shifts within the organization. By focusing on day-to-day impacts, such as implementing Honeycomb's Service Level Objectives for proactive issue resolution, they improved customer experience, reduced stress on teams, and decreased overall Mean Time to Recovery (MTTR). This success led to organizational buy-in, with Vanguard mandating the use of OpenTelemetry and Honeycomb across all applications, aiming to replace existing APM tools by the end of 2021, thereby reinforcing the broader adoption of observability practices.

Honeycomb's Lattice design system aims to enhance user experience by focusing on accessibility and intuitive design, allowing users to troubleshoot seamlessly. In response to the growing importance of accessibility, Lattice's development involved auditing and streamlining typography across the product, resulting in a simplified system with two Google Fonts, Roboto and Poppins, and fallback options like Arial and Helvetica for better usability. The design system emphasizes guidelines that ensure clarity and consistency, such as using sentence case for readability and maintaining logical heading sequences for screen readers. Additionally, the team plans to improve data visualization elements for better responsiveness and readability, reflecting their commitment to creating a more accessible and efficient user interface.

Between 2017 and 2020, Slack experienced rapid growth, which led to significant challenges in its continuous integration (CI) system, particularly with flaky tests that undermined developer trust and slowed down deployment processes. Frank Chen, a Senior Staff Engineer at Slack, addressed these issues by implementing an observability-driven approach to trace and diagnose CI infrastructure problems, significantly reducing the flaky test rate from 50% to 5%. The solution, involving tools like SlackTrace and Honeycomb, allowed for real-time analysis of trace data, enabling quick identification and resolution of issues such as slow Git checkouts and system overloads. This transformation not only improved developer velocity and confidence but also fostered a collaborative effort among teams to maintain a robust CI environment, highlighting the critical role observability played in managing and optimizing Slack's complex infrastructure.

Balancing the need for high availability with the development of new features is a common challenge for engineering teams, which can be addressed by using service level objectives (SLOs) and error budgets to prioritize tasks. This approach was exemplified by Honeycomb, which faced an incident when integrating OpenTelemetry Protocol (OTLP) support into its ingest service, resulting in burned error budgets and high response times. To rectify the situation without impacting existing traffic, Honeycomb created a separate cluster for OTLP traffic, leveraging AWS's gRPC support in their Application Load Balancers. This allowed the team to iterate and test safely while maintaining the reliability of their main services. The incident highlighted the importance of SLOs in facilitating discussions about product development and infrastructure investment, ultimately enabling Honeycomb to deploy OTLP support effectively and gather customer feedback.