February 2024 Summaries

Incident investigations and reports present numerous challenges, particularly when balancing time pressures, stakeholder expectations, and the types of investigations required. Inspired by Sidney Dekker's framework, investigations can serve moral, existential, preventative, and epistemological roles, often creating tensions when different approaches clash. Rapid analyses, often demanded by customers for public relations, may compromise the depth and learning potential of an investigation. At Honeycomb, the strategy involves producing multiple reports tailored to various audiences, such as status updates, preliminary reports, in-depth internal reviews, and redacted public reports, to maximize learning while addressing public and internal needs. This approach prioritizes learning from anomalies and large incidents, choosing depth over breadth in reviews, and adapting strategies as the company grows. Engaging with stakeholders and the broader community through platforms like Slack encourages continuous improvement and dialogue on incident management practices.

OpenTelemetry has significantly advanced observability in the software industry over the past five years by unifying logs, traces, and metrics under a vendor-neutral API and fostering collaboration among users and vendors to shape the future of telemetry. Initially, these goals seemed ambitious, but the project has successfully achieved them, offering libraries for code instrumentation in 11 languages, semantic conventions for consistent telemetry data, and a collector to decouple telemetry sources from destinations. The project's success is attributed to remarkable collaboration across end users and vendors, which has driven continuous improvements. The author, who joined OpenTelemetry in 2019 and has been actively involved in its Configuration Working Group and Security Special Interest Group, is excited to join Honeycomb as a Staff Software Engineer to further contribute to OpenTelemetry's progress and enhance user experiences, having been introduced to event logging and distributed tracing by Honeycomb in 2018.

The text explores the evolution and significance of Application Performance Monitoring (APM) within software development, detailing its journey from infrastructure monitoring to a more nuanced understanding of application performance. Initially, tools like Nagios and statsd focused on infrastructure metrics, but as software complexity increased, especially with the advent of cloud computing and containerization, the need for detailed application metrics became evident. APM now encompasses monitoring application-specific metrics such as response times, error rates, and resource usage, thereby offering insights into both applications and their underlying infrastructure. This capability helps businesses make informed decisions based on customer expectations and contractual obligations. The text also highlights the shift towards observability, or "Observability 2.0," which emphasizes customizable metrics without additional costs, fostering a deeper understanding of software performance across different business domains. This development signifies a move from standardized solutions to more tailored insights, allowing developers and operations teams to collaboratively enhance software reliability and user experience.

The text discusses the challenges and considerations in using Large Language Models (LLMs) in production, particularly focusing on the importance of data over the models themselves. It highlights the shift in machine learning from model-centric to data-centric approaches, emphasizing the necessity of relevant, sufficient, and high-quality data to make LLMs effective in practical applications. The concept of Retrieval Augmented Generation (RAG) is introduced as a method to enhance LLMs by feeding them specific data without the need for extensive model fine-tuning. The post illustrates these ideas using the Query Assistant example, which uses vector embeddings and cosine similarity to identify relevant data columns for user queries, underscoring the need for experimentation and understanding user interaction patterns. It also addresses the data relevancy, magnitude, and quality issues and the lack of existing tools to easily determine the right data needed for LLM success, while encouraging a deep understanding of product use to guide data selection and implementation efforts.

Honeycomb has introduced new Ingest-Only API Keys aimed at simplifying and securing the management of observability complexity for its customers. These specialized, environment-scoped keys are designed to securely transmit telemetry data and optionally create new datasets, with limited permissions that only allow data writing, making them immutable and safe for client-side instrumentation. They can be easily deleted or disabled to help manage compliance and reduce clutter. This development introduces a dual-key system, comprising the new Ingest Keys and the pre-existing Configuration Keys, with recommendations to use the Ingest Keys for data writing tasks due to their scoped permissions and immutability. Future enhancements will include Classic customer support and programmatic key management through a Key Management API, allowing for the creation, disabling, and deletion of keys, which are currently managed via the UI.

Data sovereignty has become increasingly important for observability teams due to complex regulations on telemetry data, which dictate what data can be captured, where it can be stored, and for how long. OpenTelemetry provides flexible solutions to manage data routing, filtering, and compliance with these regulations, alleviating the operational burden previously associated with proprietary tools or custom-built pipelines. Through the use of OpenTelemetry Collector's routing and transformation capabilities, telemetry data can be directed based on geographic origin and other attributes, ensuring appropriate handling as per local regulations. Furthermore, it offers mechanisms to filter and redact personally identifiable information (PII), thus achieving compliance through multiple layers of control. The process can be enhanced with tools like Honeycomb's Refinery, which aids in sampling and managing telemetry data to prevent PII exposure while maintaining its utility for debugging and service level objectives. Honeycomb provides support for teams to harness the full potential of OpenTelemetry, facilitating the creation of sustainable and scalable observability practices.

Honeycomb is a versatile observability platform that plays a crucial role throughout the software development lifecycle (SDLC) by providing comprehensive insights into software performance and user interactions. Its integration into the "Operate" phase allows for efficient troubleshooting through dynamic queries and trace analyses, helping identify root causes of errors and performance issues. Honeycomb enhances the implementation phase by utilizing distributed tracing to pinpoint where code changes are needed and assess their impact on other services. During testing, it consolidates data to create a unified view across teams, reducing miscommunication and expediting problem resolution. In deployment, Honeycomb alleviates the fear of unforeseen issues by enabling developers to monitor the performance and impact of their changes in real time. Additionally, it aids operations by reducing alert fatigue and focusing on customer-impacting issues, linking operations back to planning by informing product managers about how features are used and performing. Honeycomb's unique ability to provide rich context and interconnected data makes it a powerful tool that extends beyond traditional monitoring to support informed decision-making across the SDLC.

Telemetry data is a crucial tool for software monitoring and troubleshooting, allowing developers to track errors and set business goals like Service Level Objectives (SLOs). However, changes in telemetry data, such as updates in field names or values, can disrupt alerts, sampling rates, and SLOs. These changes often occur when updating libraries, such as OpenTelemetry, which recently revised its HTTP Semantic Conventions. To manage these changes, developers can use three key components in their telemetry pipeline: the Honeycomb UI, the OpenTelemetry Collector, and the sampling proxy, Refinery. By accommodating field name changes, such as transitioning from 'http.status_code' to 'http.response.status_code,' and adjusting for new field values, these tools help maintain consistent data analysis and sampling. Derived columns and transform processors can standardize fields across various environments, ensuring continuity despite evolving telemetry standards. These techniques not only aid in OpenTelemetry migrations but also in integrating nonstandard telemetry and evolving internal telemetry standards, ultimately allowing developers to maintain control over their telemetry processes.