June 2022 Summaries

Honeycomb offers a unique approach to Application Performance Monitoring (APM) with a pricing model based solely on event consumption, allowing significant cost reductions compared to traditional APM tools that charge based on various factors like memory or user seats. By consolidating logging, tracing, and metrics into a single platform, Honeycomb eliminates the need for separate tools and their associated costs. Its compatibility with OpenTelemetry helps avoid vendor lock-in, and it supports high-cardinality data without additional charges, providing detailed insights without compromising performance. Honeycomb's unlimited user access to features further enhances its value, making it an attractive option for teams looking to optimize their monitoring expenses and improve observability practices.

In the rapidly evolving landscape of modern distributed systems, achieving effective system management increasingly relies on distinguishing between monitoring and observability. While traditional monitoring focuses on evaluating specific system metrics against predefined thresholds to identify known issues, it often falls short in complex environments where system failures are unpredictable. Observability, on the other hand, emphasizes the ability to ask arbitrary questions about a system's state using high cardinality, high dimensionality, and exploratory data, allowing for real-time analysis and troubleshooting without prior knowledge of potential issues. As highlighted in the Authors’ Cut webinar that delves into chapters of the new book "Observability Engineering Achieving Production Excellence," observability is essential for understanding the internal state of complex systems and providing a framework to address unknown unknowns. The book, co-authored by Liz Fong-Jones, George Miranda, and others, explores the nuances between monitoring and observability, showcasing how the latter provides deeper insights into system behavior, particularly in the context of debugging and troubleshooting. The discussion underscores that while monitoring is suitable for simpler systems with predictable failures, observability is crucial for navigating the intricacies of modern software environments.

Honeycomb leverages AWS Lambda for its query execution, benefiting from its scalability and pay-per-use model to maintain speed and affordability. However, understanding the costs associated with specific customers or application areas can be challenging. By utilizing a derived column approach, Honeycomb connects customer experiences to AWS Lambda costs, helping identify cost-intensive usage patterns and analyze the financial impact of software changes. AWS Lambda's pricing is based on architecture, RAM size, and invocation duration, with costs calculable through a linear formula. Honeycomb uses in-app instrumentation to capture detailed data, which includes Lambda architecture, configuration, and customer-specific identifiers, enabling more precise cost analysis. This method assists in validating customer usage expectations, setting up alerts for unexpected expenses, and conducting experiments to optimize Lambda configurations, ultimately achieving significant cost savings. The derived column translates compute units into monetary values, facilitating discussions about optimization versus feature development and helping determine when further optimization is worthwhile. Honeycomb encourages experimentation with this method to gain deeper insights into resource costs and customer behaviors.

Paige Cruz, a retired Site Reliability Engineer (SRE), shares her journey from relying on Application Performance Monitoring (APM) tools to embracing observability after experiencing a critical on-call incident that revealed the limitations of her existing tools in a modern, distributed system. Despite initial resistance due to cost and the learning curve associated with new tools, Cruz illustrates how observability provides comprehensive insights into system behavior, enabling faster debugging and more informed responses to incidents. She challenges common misconceptions about observability, such as its perceived expense and complexity, by highlighting its benefits over traditional APM tools, including its ability to handle high-cardinality data without additional cost and its capacity to equip even novice engineers with the information needed to resolve issues efficiently. Through her experience, Cruz emphasizes that observability is not merely an additional expense but a crucial component for managing the intricacies of contemporary cloud environments, ultimately leading to more effective and less stressful on-call experiences.

Debugging application performance in Azure AppService presents challenges with Azure's built-in tools, prompting the use of Honeycomb for near real-time analysis of HTTP Access Logs. The process involves ingesting logs from Azure AppService, which operates with a built-in Load Balancer, for performance investigation, leveraging Azure Event Hubs for streaming data, and utilizing Honeycomb for exploratory analysis. Azure AppService, a platform for Managed Application Hosting, supports various programming languages and abstracts infrastructure concerns from developers. HTTP Access Logs provide insights into site usage, such as client IP addresses and request durations, which can help in understanding user experience and identifying performance issues. The setup requires creating an Azure Event Hub Namespace, enabling diagnostic logs to stream into the Event Hub, and deploying a FunctionApp to forward logs to Honeycomb. This approach offers a cost-effective method for enhancing visibility and analysis of AppService performance, with plans for future posts to expand the methodology to other Azure services.

The software industry is increasingly adopting the model of service-ownership teams, where teams are responsible for the entire lifecycle of the software they build, a shift supported by methodologies like microservices, DevOps, Agile, and Project to Product. Honeycomb supports this paradigm by offering tools that provide visibility into production, enabling teams to monitor and manage their services effectively. By organizing data around service ownership and utilizing distributed traces, Honeycomb allows teams to understand the interconnections between services and the full context of customer requests. The platform separates events by environment, maintaining clarity between production and test data, and adheres to the OpenTelemetry standard to provide detailed insights. Honeycomb's approach aligns with modern observability practices by dividing events at the service level while allowing for comprehensive analysis across services, thus enhancing the capability of teams to ensure smooth operation and resolve issues at the interfaces between services.

Observability is defined as the ability to understand the internal state of systems based on their telemetry, aiding in troubleshooting, debugging, and performance tuning, which is often misunderstood as merely a collection of logs, metrics, and traces. Honeycomb advocates for using structured events as the foundation for observability, enabling the visualization of trends and patterns through traces composed of these events, offering a more comprehensive view than traditional methods. The approach emphasizes the use of arbitrarily wide structured events, which provide detailed data for effective debugging and analysis, as opposed to the fragmented insights from metrics, logs, and traces. Honeycomb's platform facilitates the storage and querying of raw data, preserving context and granularity, and supports distributed tracing to track requests across microservices, enhancing the understanding of system performance and failure points. The use of OpenTelemetry for instrumentation is recommended to generate telemetry data, avoiding vendor lock-in and ensuring compatibility with various backend stores, including Honeycomb, as discussed in their webinar series and O’Reilly’s "Observability Engineering" book.

Being on call for the first time can be daunting, but with proper preparation and support from your team, it can become a manageable experience. Essential steps include understanding written guidelines, clearing your project schedule, and familiarizing yourself with a documented response SLA. Engaging in business hours on-call practice and shadowing or reverse shadowing with experienced colleagues can build confidence. If your service lacks frequent incidents, participating in game days or chaos engineering exercises can simulate on-call scenarios. Strengthening skills with observability tools, like Honeycomb, is crucial before actual incidents occur. Always having a primary and secondary on-call partner ensures support and aids in effective escalation, which should be seen as an opportunity to learn rather than a failure. Resources like the Google SRE book and mentorship from experienced colleagues can further enhance your understanding and readiness for on-call duties.

Phillip, a product manager at Honeycomb, shares insights into the complex world of observability, highlighting its intricate terminology and the challenges faced in understanding and implementing it effectively. Observability, often confused with traditional monitoring, involves collecting and analyzing data like traces, metrics, and logs to gain insights into cloud-native applications. Phillip emphasizes the importance of querying data, often overlooked by vendors, to diagnose performance and reliability issues, while noting the current reliance on human pattern recognition over machine learning for identifying system problems. He advocates for OpenTelemetry as a promising standard for data collection, despite its nascent status, and underscores the need for easier code instrumentation akin to making code debuggable. While observing that achieving observability can be daunting, Phillip remains optimistic about its potential, especially in organizations with high observability maturity, and encourages further exploration of its applications in testing and development environments.

In June 2022, Honeycomb was recognized as a Leader in the Magic Quadrant for Application Performance Monitoring and Observability by Gartner, marking the first time observability has been acknowledged as a distinct category in this context. Honeycomb's platform, which differs from traditional Application Performance Monitoring (APM) tools, enables high-performance engineering teams to visualize, analyze, and improve the quality and performance of cloud applications by providing detailed insights into how code behaves with real users. This capability allows teams to quickly diagnose issues, improve uptime, and enhance service performance, ultimately leading to better customer experiences and more time for innovation. Honeycomb's recognition reflects its ability to address the complexities of cloud-native environments where traditional APM tools fall short, offering a new approach to understanding system states and solving engineering challenges efficiently.

Honeycomb celebrates its recognition as a Leader in the 2022 Gartner Magic Quadrant for Application Performance Monitoring (APM) and Observability, highlighting the evolution of observability as distinct from traditional APM. Observability is described as an exploratory, analytics-driven approach that empowers users to investigate and customize their queries, differing from APM's out-of-the-box solutions. The company emphasizes the limitations of traditional monitoring tools, such as alert fatigue and dashboard blindness, and underscores the business and technical benefits of adopting observability, including reduced downtime and developer churn. Honeycomb's commitment to making observability accessible to all engineering teams is reflected in its free tier and dedication to fostering curiosity and exploration, aiming to improve software systems' understanding and debugging. The recognition by Gartner and support from its community, investors, and team are seen as vital to Honeycomb's mission to refine and lead in the observability space.

Microsoft's decision to integrate Activity into the .NET Base Class Library (BCL) since .NET Core 2.0 has created some confusion, particularly with the emergence of OpenTelemetry (OTel) which uses different terminology. Activity serves as a tool for monitoring and telemetry within .NET, predating OpenTelemetry and providing easy adoption due to its integration into the BCL. However, the OpenTelemetry .NET Shim was developed to bridge the terminology gap, allowing developers to use OpenTelemetry's Tracer and Span concepts while leveraging Activity and ActivitySource under the hood. This shim is useful for maintaining consistent terminology across different programming languages but may not be necessary for developers working exclusively within .NET. The recommendation is to use Activity for libraries intended for wider distribution due to compliance considerations, while using OpenTelemetry directly offers advantages in applications with full control due to its comprehensive features and plugins. Looking ahead, there is a hope for the BCL to adopt a consistent tracing pattern similar to its approach with logs and metrics, relying more on open-source solutions like OpenTelemetry rather than integrating additional features directly into the framework.

The text provides a guide to getting started with observability using OpenTelemetry and Honeycomb, emphasizing that observability can be beneficial on a small scale and individually, without requiring a large company-wide initiative. It outlines a three-step process: setting up a local environment to send data, configuring the application to transmit events using OpenTelemetry libraries, and analyzing the resulting data to gain insights into the software's behavior. The guide offers specific instructions for Java and Node.js applications, while encouraging users to explore the documentation for other languages. It highlights the value of visualizing data to discover insights and potential surprises about software performance, introducing the concept of "Mean Time To WTF" as a measure of successful tracing. The ultimate goal is to use these insights to improve development and potentially advocate for broader implementation across teams and in production environments. Additionally, it invites users to seek help and engage with the team behind Honeycomb through workshops, office hours, and other communication channels.

Honeycomb has expanded its Terraform Provider to include support for managing Service Level Objectives (SLOs) and Burn Alerts, enhancing its configuration-as-code capabilities and allowing users to manage these features programmatically. This development follows the release of the Honeycomb Management API in 2021, which facilitated the creation of a community-driven Terraform provider. The new version 0.7.0 of the Honeycomb Terraform Provider includes resources such as honeycombio_slo and honeycombio_burn_alert, enabling seamless integration of SLO management into existing Terraform workflows. These updates allow for the configuration of derived columns as Service Level Indicators (SLIs) and facilitate the automation of error budget alerts, empowering users to apply best practices such as version control and peer review to their SLO lifecycle management. Honeycomb SLOs, which are available to Pro and Enterprise users as of April, help reduce alert fatigue and ensure reliable and actionable alerts, with the Terraform Provider now accessible in the Terraform Registry.

Seven years after Rent the Runway popularized open-sourcing engineering ladders, many organizations, including Honeycomb, have re-evaluated their approach to job promotions, which traditionally tied advancement to the "area of scope." At Honeycomb, there's a growing belief that valuing scope alone creates unfair incentives, as it rewards engineers who undertake the largest projects without necessarily addressing the company's most crucial problems. To counteract this, Honeycomb introduced a new career progression framework called PSHE (problem, solution, how, and execution), originally developed by YouTube's product team, which emphasizes both scope and ownership. This new framework allows for career growth not solely based on project size but also on problem-solving and execution capabilities. Honeycomb's revised job ladder, although not yet open-sourced due to copyright complexities, now incorporates visual tools to illustrate its principles, focusing on growth through both scope and ownership dimensions to reflect company values and support engineers' development.