February 2023 Summaries

At Honeycomb, managing incidents in a rapidly growing system involves acknowledging that some issues cannot be preemptively addressed and instead require strategic responses when they arise. The organization recognizes incidents as an inevitable part of operating complex systems, where past decisions interact with evolving demands, and has developed a culture of psychological safety and shared responsibility, empowering on-call responders to act swiftly and escalate when necessary. Incident management at Honeycomb eschews rigid frameworks in favor of flexible coordination patterns, such as creating dedicated communication channels and utilizing lightweight tools, which facilitate information sharing and collaborative problem-solving across teams. The emphasis is on maintaining alert hygiene to reduce cognitive load during high-pressure situations, promoting transparency, and using incidents as learning opportunities to adjust processes and enhance future resilience. Honeycomb's approach is dynamic, with priorities and methods evolving to adapt to new challenges while maintaining a focus on core features and critical functionality.

Honeycomb developed a dynamic service map from tracing data to provide a real-time, interactive visualization of system architecture, aiming to enhance observability workflows. Unlike static maps, this innovative tool reflects ongoing changes in the system, aids real-time issue investigation, and helps users formulate queries using Honeycomb's query engine. The project faced challenges in design and engineering due to varying data sources and system complexities, prompting the team to create a script for simulating real data, which allowed for iterative design and testing. Early user feedback was crucial in refining the map's layout and functionality, ultimately leading to a successful launch in October 2022. The service map has since become a valuable tool for onboarding, incident response, and identifying gaps in system instrumentation, exceeding initial expectations for its application.

In an innovative move, Honeycomb initiated an experiment in 2022 to include an employee as a full voting member on its board of directors, an approach uncommon among venture-backed companies. This experiment, aimed at promoting transparency and diversity, involved an internal election where eligible employees could run for the position, resulting in Paul Osman being elected, who later passed the role to the author after leaving the company. The employee board member, who participates in board meetings and closed sessions, serves as a bridge between employees and the executive team, offering insights from a non-executive perspective and enhancing organizational confidence by connecting various departmental strategies. The experiment has highlighted the complexities and learning curves involved in board participation, while also underscoring the value of such representation in understanding broader company strategies and fostering internal communication. As Honeycomb extends this initiative for another three years, an internal committee is working on refining the election process and term duration to ensure fair representation across the company, with the aim of maintaining and enhancing employee involvement at the board level.

Setting Service Level Objectives (SLOs) is advocated as a transformative practice for engineering teams, offering benefits such as improved sleep for engineers, reduced friction between engineering and management, and cost-effective monitoring. SLOs, defined as reliability targets based on Service Level Indicators (SLIs), help ensure smooth user experiences and compliance with Service Level Agreements (SLAs), serving as early warning systems for potential performance issues. However, they must be implemented judiciously, focusing on meaningful, measurable, and realistic targets to be more effective than traditional alerting methods. The success of SLOs hinges on alignment across teams, from developers to business units, and their integration with key business metrics and user journeys. By fostering a shared language and understanding of product performance, SLOs bridge the gap between engineering and business perspectives, enhancing incident resolution and compliance. Additionally, coupling SLOs with observability can further enhance their effectiveness, reducing alert fatigue and equipping on-call engineers with better tools for rapid problem resolution.

Honeycomb's newly launched Service Map feature offers users the ability to quickly understand the complex interconnectivity of services within intricate environments by allowing them to filter by services and traces, thereby isolating specific areas for analysis. This feature is designed to replace outdated architecture diagrams and reduce reliance on individual engineers' knowledge by providing a dynamic and interactive tool that helps surface issues and understand system evolution. Service Map is available as a separate scenario in Honeycomb's Sandbox, offering a real-world use case where users can isolate and inspect services, apply advanced filters for debugging, and investigate incidents such as slow requests. It integrates with other Honeycomb tools like BubbleUp and trace waterfalls, providing a comprehensive introduction to its capabilities and encouraging users to explore it with their own data.

When organizations adopt Honeycomb at the Enterprise level, they receive support from a Technical Customer Success Manager (TCSM) to help implement and leverage the observability tool effectively. The TCSM assists central observability teams in developing strategies to ensure their colleagues can use Honeycomb, addressing basic needs like login access, data availability, and query training, and working collaboratively with Honeycomb's Customer Success department to solve problems. The overarching goal is to foster a customer experience (CX)-centric culture where customer experience dictates product development decisions. However, challenges arise when engineers new to operations roles or unfamiliar with tools like distributed tracing face high-pressure situations, such as late-night outages, without sufficient practice time. Observability teams must navigate organizational constraints by identifying and utilizing existing social institutions and routines, such as team onboarding, sprint tasks, and deployment watch parties, to build engineers' confidence and promote observability-driven development. By integrating Honeycomb into these familiar activities, engineers can gradually enhance their understanding of the system, leading to better incident response and system optimization. Honeycomb's Customer Success team supports this organizational shift by providing resources such as online training and documentation while collaborating with teams to tailor strategies that maximize the tool's benefits.