March 2020 Summaries

Honeycomb's Service Level Objective (SLO) feature was designed to enhance service reliability by providing a mechanism to measure and monitor service quality. The feature evolved with insights from Liz Fong-Jones, an experienced Google SRE, and leveraged Honeycomb's ability to store rich data, enabling unique SLO capabilities. The development process revealed several challenges, such as the need for an intuitive monitoring experience over the creation process and the importance of an alerting system to warn users of potential SLO failures. Early user feedback highlighted the feature's ability to identify key issues, although initial enthusiasm waned due to the absence of alerts. Honeycomb's alert system was refined after a costly AWS incident, emphasizing the balance between accuracy and system efficiency. The experience underscored the significance of volume in SLOs, the importance of testing pathways instead of individual users, and the need for continuous iteration to refine SLOs. Honeycomb's journey with SLOs serves as a guide for others in implementing these observability features, demonstrating the complexities and insights involved in rolling out such a system.

OpenTelemetry, a project of the Cloud Native Computing Foundation (CNCF), is a significant advancement resulting from the merging of the OpenTracing and OpenCensus projects, providing a unified solution for instrumenting code for metrics and traces without being tied to any specific tool or vendor. Honeycomb has enthusiastically embraced OpenTelemetry, highlighting the new Honeycomb exporter for the OpenTelemetry Collector, which allows users to send trace data from various services like Jaeger, Zipkin, and OpenCensus to Honeycomb with no code changes. This collector can process and export data in multiple formats and can be deployed as an agent, sidecar, or standalone service, offering a seamless way to leverage existing instrumented code. Honeycomb's Python SDK exporter, alongside its Go counterpart, offers an alternative for those who prefer not to run the collector, enabling direct telemetry data export with minimal code adjustments. As OpenTelemetry approaches its beta phase, Honeycomb is focusing efforts on this exporter, while still supporting existing tools, and is expanding language support, including a .NET exporter contributed by the community, allowing broader adoption of vendor-neutral instrumentation with Honeycomb.

Amazon's announcement of the AWS Graviton2 processor, designed to power M6g instances, offers a promising alternative to traditional x86-powered instances in terms of performance and cost, particularly for workloads built on the aarch64 architecture. The transition to Graviton2 requires recompilation of applications reliant on x86, but experimentation by Liz Fong-Jones at Honeycomb.io demonstrated significant benefits, such as lower costs and improved performance in specific use cases. Despite challenges in adapting to the new architecture, including recompiling dependencies and addressing compatibility issues, the effort resulted in a 30% reduction in instance usage and a 10% cost saving compared to current C5 instances, along with enhanced performance metrics. Although M6g instances are not yet widely available for production, the initial transition yielded compelling results, indicating potential long-term benefits once these instances are commercially accessible. Honeycomb.io's experience underscores the strategic advantages of adopting Graviton2, anticipating further integration as availability improves, while continuing to rely on C5 instances in the interim.

Honeycomb, a company focused on observability, marked 2019 as a pivotal year in solidifying observability as a recognized category and expanding their market presence, which led to the hiring of Chad Malchow as the new VP of Sales. Chad's experience in driving hypergrowth at GitLab and building sustainable strategies at Return Path is expected to enhance Honeycomb's sales team, strategy, and processes, allowing more engineering teams to benefit from observability solutions. His ability to connect with engineering teams and understand their challenges is anticipated to foster strong, long-term customer relationships. As a distributed executive, Chad's role is particularly significant as Honeycomb adapts to a work-from-home model, contributing to the company’s evolution into a distributed organization and aligning with its core values and hiring philosophy. The leadership team, including board member Ariel Tseitlin, is enthusiastic about Chad's potential to scale the business and build on Honeycomb's existing strengths.

The concept of "sociotechnical systems" highlights the intertwined nature of social and technical problems, emphasizing that neither can be fully addressed in isolation. Observability-driven development (ODD) is presented as a transformative approach in software engineering, enabling teams to write better code by providing real-time insights into system operations. The text argues that ODD fosters a more productive, empowered, and less stressed workforce by allowing developers to interactively explore live production environments, thereby reducing technical debt and enhancing system reliability. The author suggests that while observability is a critical component of modern software development, its full potential is realized when combined with social changes, such as fostering team collaboration, encouraging diverse viewpoints, and promoting shared value systems. The narrative acknowledges the industry's current inefficiencies and proposes that adopting ODD could lead to a significant shift, making software delivery less daunting and more efficient, ultimately benefiting not just elite teams but everyone involved in software development.

Emily Nakashima's promotion to VP of Engineering at Honeycomb highlights the company's commitment to internal advancement and transparency in leadership values. Initially hired as an intermediate engineer, Emily rapidly progressed through several roles due to her dependability, leadership, and engineering excellence. Her journey reflects Honeycomb's belief in nurturing talent from within rather than hiring externally, emphasizing trust and admiration for existing employees' growth potential. Emily's leadership style, characterized by mission-driven actions, dependability, and constructive feedback, has fostered a collaborative and productive work environment. Despite initial challenges, Emily's ability to adapt and the strong relationships she has built within the company underscore her suitability for the role. Honeycomb acknowledges the risk of losing talented executives like Emily but remains committed to ensuring that employment at the company remains the best career option for its employees as long as possible.