November 2023 Summaries

Observability in production environments is crucial for understanding system behavior, yet it is often neglected or added retrospectively, which can result in a loss of context. The text proposes integrating observability into the development process similarly to how tests are treated, ensuring that instrumentation is added as features are developed, thereby maintaining context and enabling real-time monitoring. OpenTelemetry is recommended as a standard for tracing applications, which can enhance logging by showing correlations and relationships between operations. The concept of automatic instrumentation is introduced, allowing developers to instrument code without altering it, while manual or custom instrumentation can provide more detailed insights by adding specific attributes or creating new spans within the code. The combination of these approaches helps developers gain confidence in their systems by confirming expected changes and predicting production behavior, ultimately facilitating easier debugging. Emphasizing observability as part of the development workflow can transform it from a “nice to have” into a standard practice, akin to writing tests.

Since launching Query Assistant, Honeycomb has gained insights into managing Large Language Models (LLMs) in production, exploring new techniques that they share to improve LLM applications. They emphasize the complexity of operating LLM-based apps due to the unpredictable nature of user interactions and the nondeterministic outputs of LLMs. Honeycomb advocates for using observability tools, like distributed tracing and Service Level Objectives (SLOs), to identify and address performance issues in real-time, which is crucial for maintaining reliable LLM applications. They highlight the importance of creating evaluation systems and a feedback loop based on real-world usage data to enhance system reliability and accuracy. Honeycomb acknowledges the challenges of deploying LLMs in production, such as managing "hallucinations" or unintended outputs, and suggests that actively monitoring trace data can mitigate these issues. They support the OpenLLMetry project for its potential to simplify the integration of observability practices in LLM applications and are developing new product capabilities to facilitate these processes. Honeycomb encourages users to explore their documentation and resources, including an O’Reilly report, to implement observability in their LLM endeavors.

In a landscape where modern, agile working environments are often seen as the ideal, this text discusses practical approaches to dealing with software issues in less-than-ideal settings. It emphasizes the importance of implementing telemetry to identify and report errors accurately, which can help pinpoint whether an application or a combination of downstream systems is the source of a problem. The piece highlights the challenges of legacy systems, such as incorrect HTTP status codes that can mislead error attribution and stresses the need for defensive measures like adding attributes to spans for better traceability and accountability. It also covers the collaborative benefit of sharing insights across teams to improve system reliability and suggests using tools like OpenTelemetry to enhance traceability. Moreover, the text encourages organizations to address common pitfalls by employing defensive retry logic and highlights the importance of communication and teamwork in resolving service dependencies. It ultimately advocates for a proactive approach to error handling and system maintenance, suggesting that this can lead to clearer accountability and more efficient problem resolution across teams.

The introduction of budget rate alerts in Honeycomb's Service Level Objectives (SLOs) enhances the alerting system by addressing the limitations of traditional SLOs, which often forced users to choose between focusing on operations or decision-making. Budget rate alerts provide notifications based on the rate of budget consumption, allowing users to be alerted of issues even if the budget has been exhausted, thus offering a more balanced approach to both operational and strategic needs. This feature improves incident response by enabling users to quickly identify and prioritize genuine problems while avoiding false alarms. It also aids in organizational decision-making by providing insights into budget usage patterns, helping to anticipate outages or changes in usage that require strategic adjustments. The interactive configuration allows users to set alerts efficiently, ensuring they are informed about service degradation promptly. The feature is available for Enterprise and Pro customers, with options for trials and easy onboarding for new users.

Kubernetes presents challenges for developers in understanding and managing applications due to its complex infrastructure, where both the cluster state and application health influence each other. Many developers struggle with inadequate tools that focus primarily on low-level metrics, leaving them unable to effectively correlate these metrics with application behavior or diagnose issues. The gap in telemetry collection and management often results in frustration, especially when trying to troubleshoot problems across vast numbers of pods. OpenTelemetry offers a solution by providing a comprehensive framework for capturing and transforming telemetry data, allowing developers to enrich application and service telemetry with critical Kubernetes metadata. This enables improved correlation and analysis, reducing the risk of vendor lock-in and helping developers tailor their observability pipelines. Despite OpenTelemetry's potential, developers must still develop practices for analyzing the data and understanding their systems better. Honeycomb is introducing new integrations to aid developers in navigating these complexities, offering insights into Kubernetes-based applications and promoting active use of OpenTelemetry for enhanced visibility and control.

In the blog post, the author delves into the complexities of configuring telemetry pipelines using OpenTelemetry Collectors, specifically addressing the challenges of managing headers for data signals from diverse sources like Kubernetes and Java virtual machines. The post introduces the Headers Setter extension as a solution to streamline the process of passing necessary headers, such as X-Honeycomb-Team and X-Honeycomb-Dataset, through telemetry pipelines without complicating application configurations. By using this extension, users can ensure that specific headers accompany the OpenTelemetry signals through the pipeline, facilitating correct data routing and authentication. The author provides detailed configuration guidance for receivers, exporters, and batch processors within OpenTelemetry Collector setups, offering practical examples to demonstrate how to maintain metadata integrity and control the flow of telemetry data. The blog concludes with insights into setting up environment variables for applications to manage header configurations effectively and highlights a new feature, Honeycomb for Kubernetes, aimed at enhancing observability in Kubernetes environments.

Honeycomb has introduced Honeycomb for Kubernetes, a new observability feature designed to enhance the debugging and migration processes for engineering teams by correlating code performance with granular cluster data. This tool aims to bridge the gap between application and infrastructure layers, which often causes operational backlogs and decreased productivity due to inadequate context in existing application performance monitoring (APM) suites. By providing Kubernetes-aware observability, Honeycomb offers developers and platform engineers the ability to correlate application requests with specific Kubernetes configurations, simplifying the identification of issues and improving developer productivity. With integration options, including OpenTelemetry and a low-code agent, Honeycomb supports a scalable and efficient approach to observability, accommodating various platforms such as Amazon EKS, Azure Kubernetes Service, and Google Kubernetes Engine. This development reflects Honeycomb's commitment to providing comprehensive insights into complex systems, thus enabling engineering teams to release updates more confidently and maintain seamless migrations.

Honeycomb for Kubernetes is a newly launched solution aimed at bridging the gap between infrastructure and application teams by offering an integrated observability tool that combines Kubernetes metrics, logs, events, and application traces with precise metadata. Available at no additional cost, it addresses the complexities of Kubernetes environments by providing a unified view that eliminates the fragmented approach of traditional monitoring tools. This solution leverages OpenTelemetry to offer quick and easy instrumentation, allowing teams to trace issues across both infrastructure and applications with advanced correlation workflows using Honeycomb's query engine and BubbleUp feature. Enhanced with AI-powered query capabilities and curated dashboards, it simplifies diagnostics and fosters effective collaboration, complementing existing tools like Prometheus and Grafana. Honeycomb for Kubernetes empowers teams to operate from a shared truth, reducing blame games and uncertainty while offering low-code and no-code options for those facing constraints, ensuring comprehensive visibility and a seamless observability experience.