June 2023 Summaries

This text offers a detailed exploration of how to conduct a proof-of-concept (POC) for Honeycomb and OpenTelemetry over four weeks, assuming unlimited resources. It begins with setting up OpenTelemetry, an open-source project for standardized telemetry collection, and Honeycomb, an observability tool that processes high-dimensional events for system insights. The first week focuses on instrumentation through auto and manual methods, followed by setting up an OpenTelemetry Collector and exploring Honeycomb's features like the Query Builder and BubbleUp. The second week involves enhancing the Collector, setting up alerts and integrations, and engaging developers for targeted instrumentation. The third week emphasizes adding deployment markers, leveraging stakeholders for defining Service Level Objectives (SLOs), and creating derived columns for custom data fields. The final week involves reviewing system incidents, conducting user interviews, assessing telemetry producers, and setting a roadmap for future observability improvements. Throughout, the document highlights the importance of fostering a culture of observability and shared responsibility within engineering teams.

Honeycomb's experience with two seemingly identical outages highlights the nuanced nature of incident management and the importance of learning from each event. The first incident in December 2021 involved a significant disruption during their EC2 to EKS migration, as AWS SSM failures led to a prolonged outage in the us-east-1 region, prompting improvisational solutions to maintain operations. Despite the complexity and rarity of this event, the team focused on examining their adaptive responses rather than implementing specific preventative measures. In September 2022, a similar issue occurred, but the team's prior experience allowed for a more organized and efficient response, as they quickly identified the problem and leveraged previous investigations to mitigate the impact. This time, they introduced new strategies such as setting up configuration mirrors and automating region-specific solutions, demonstrating that while no two incidents are truly identical, accumulated knowledge and experience can significantly alter the management and outcome of subsequent incidents.

OpenTelemetry and Beelines are designed to trace synchronous activities, such as web application and API calls, based on the assumption that transactions begin with a user request and end with a response. While these assumptions work for most traffic, there are application flows that deviate significantly, leading to "exotic" trace shapes, such as batch processing, stream processing, and machine learning activities, which do not fit neatly into the typical request-response pattern. These types of traces can be continuous or excessively long, making it challenging to find value in them during aggregate analysis or visual representation. The text explores various strategies to adapt these non-conforming traces into more manageable and meaningful forms, such as breaking traces based on time or item count, starting new traces for each job, or using summary spans to encapsulate job activities. By managing trace identifiers strategically, organizations can apply sampling approaches to optimize the observability of their systems while minimizing data volume and cost, particularly with tools like Refinery and Honeycomb. The text also offers code examples and considerations for handling different trace shapes, emphasizing the flexibility and necessity of adapting tracing practices to diverse application patterns.

Observability-driven development (ODD) is a methodology that encourages developers to prioritize observability in their workflow, ensuring that software functionality is accurately expressed through trace data. While it is beneficial to integrate observability considerations early in the development process, the author argues against allowing observability to solely drive development, emphasizing that it should support rather than overshadow other critical aspects like customer interface, database management, and domain boundaries. The article highlights the importance of using observability as a tool to enhance development practices by providing insights into system performance and user interaction, thereby ensuring that features function as intended both during development and in production. By maintaining focus on delivering valuable capabilities to customers, developers can utilize observability to refine their architecture and troubleshoot effectively, without allowing it to detract from the primary goal of creating useful software.

Observability is crucial for effectively managing large language models (LLMs) in production due to their unpredictable and nondeterministic nature. The text discusses the challenges faced by Honeycomb while implementing their natural language querying interface, Query Assistant, and emphasizes the importance of observability to monitor LLM behavior. It highlights how subtle changes in prompts can lead to unexpected outcomes, making it essential for engineers and organizations to track errors, latency, and user feedback to identify and fix issues. Observability involves using tools like OpenTelemetry for tracing and custom attributes to gather comprehensive data on LLM interactions, enabling teams to analyze trends, understand user behavior, and optimize performance. Honeycomb's approach includes setting Service Level Objectives (SLOs) to monitor latency and error rates, which helps prioritize improvements and ensure a reliable user experience. The text underscores the necessity of observability for continuous improvement and effective management of LLM-driven features.

Migrating to Kubernetes from Amazon EC2, orchestrated via Terraform and Chef, allowed Honeycomb to better scale and manage their infrastructure, aligning with the increased adoption of containerized applications predicted by Gartner. Despite the operational complexities introduced by Kubernetes, the migration was driven by the need for a more reliable and scalable system, as well as to gain firsthand experience with a technology heavily utilized by their customers. The process involved tackling the most challenging components first, seeking external expertise for specific tasks, and utilizing Honeycomb's own observability tools to ensure seamless transitions and solve unexpected issues. The migration not only improved system efficiency, particularly seen in the reduction of Shepherd's scale-up time from 15 minutes to 2 minutes, but also provided critical insights into building safer systems and addressing scaling challenges, which were previously problematic.

In the technology sector, there's a growing emphasis on sustainability and cost management, particularly concerning infrastructure and observability tools, which are often viewed as expensive yet critical components. The suggestion to allocate 20-30% of infrastructure budgets to observability is posited to maintain a healthy environment, though this can be challenging if desired capabilities exceed that budget. Cost-saving strategies in observability include optimizing trace locality to minimize network bandwidth costs, using head and tail sampling methods to manage data processing and storage expenses, and leveraging peering and transit gateways to reduce data transfer costs across networks. Additionally, working with observability vendors and using solutions like Honeycomb's Refinery can further decrease cloud expenditures by supporting private links and enabling efficient tail sampling.

Honeycomb leverages its own platform to gain insights into its Kubernetes clusters managed through Amazon EKS, focusing on monitoring as a consumer of a managed service. One challenge addressed is detecting crashlooping pods, a difficult task with metrics-based approaches, which Honeycomb solves by using kspan to translate Kubernetes events into OpenTelemetry traces, enabling reliable detection and identification of problematic pods. Additionally, Honeycomb monitors AWS instance types to ensure optimal performance, utilizing the honeycomb-kubernetes-agent to detect when older instances are used, allowing for quick adjustments to prevent incidents. The OpenTelemetry Collector aids in monitoring the Horizontal Pod Autoscaler (HPA) by providing reliable metrics on desired pod counts, mitigating issues that arise from metrics spiking during deployments. Honeycomb's capabilities, especially in high-cardinality metrics support and event-based alerts, make it well-suited for Kubernetes monitoring, offering insights that facilitate proactive management and incident prevention.

Running a Kubernetes cluster involves complexities that can be better managed by accessing the comprehensive telemetry data that Kubernetes provides. The OpenTelemetry Collector is a recommended tool for receiving, processing, and exporting this data, as it is vendor-agnostic and modular, allowing for customization based on specific needs. Key components maintained by the OpenTelemetry community facilitate the collection of logs, host metrics, kubelet metrics, and other Kubernetes resources, though configuring these components manually can be complex and time-consuming. The OpenTelemetry Collector Helm chart simplifies this process by managing deployments within Kubernetes, allowing users to configure the necessary components and Kubernetes-specific requirements with ease. By using Helm, users can deploy the Collector with yaml configuration files that specify whether to run the Collector as a DaemonSet or a Deployment, depending on the telemetry needs of their cluster. This setup allows for efficient data collection and exportation to platforms like Honeycomb, enhancing observability within Kubernetes environments.

Martin and Jess's conversation with Todd Gardner on the O11ycast podcast highlights the complexities of frontend observability, emphasizing the importance of customer-centric observability to enhance user experience. Traditional observability methods, relying on logs and metrics, often miss the mark if they don't address the direct impact on users, especially when errors or slowdowns occur. While numerous tools like Google Analytics and performance metrics provide insights into frontend issues, they often add complexity without necessarily improving user experience. The conversation stresses that understanding the real impact on users is crucial, as demonstrated by differences in user behavior when faced with similar performance slowdowns on different pages. The discussion also touches upon the evolving nature of frontend observability, with references to OpenTelemetry and client-side instrumentation, encouraging listeners to explore further through additional resources and writings.