May 2023 Summaries

Companies often delay necessary migrations or technical upgrades due to fears of potential disruptions, but observability can be a crucial tool to mitigate these risks. Honeycomb's observability platform helps teams by providing visibility into system behavior before, during, and after migrations, allowing for quick identification and resolution of issues. This approach is particularly valuable in complex migrations, such as those involving shifts from monolithic to microservices architectures or from on-premises to cloud environments. By using tools like Honeycomb's Service Map, proactive Service Level Objectives (SLOs), and advanced querying capabilities, organizations can plan and execute migrations effectively, minimizing downtime and improving overall system reliability. A case in point is CCP Games, which successfully transitioned its architecture with the help of Honeycomb, maintaining seamless user experience even during extensive changes. Observability not only aids in smooth transitions but also ensures ongoing system performance by facilitating fast incident response, debugging, and post-migration validation, ultimately transforming migrations from daunting tasks into manageable processes.

Observability has emerged as a crucial component in modern software development, emphasizing the need to integrate it into the pre-deployment stages to enhance code quality and reduce post-deployment issues. By incorporating observability early, teams can proactively identify and resolve potential problems, thus minimizing customer impact and improving developer satisfaction. Companies like HelloFresh have utilized tools like Honeycomb to streamline incident response and adopt proactive strategies using service-level objectives (SLOs), tracing, and service maps. This approach, termed "observability-driven development," encourages developers to focus on uptime and availability from the outset, facilitating data-driven decisions and real-time insights into code functionality. By fostering a culture where observability is integral rather than an afterthought, development teams can maintain productivity and creativity while ensuring robust system performance and minimizing disruptions from new code deployments.

Traceloop addresses the challenge of writing tests for complex systems by leveraging large language models (LLMs) to interpret OpenTelemetry traces and generate tests that ensure systems function as intended. The process involves clustering incoming traces to select representative samples, building a semantic model of the system, and using this model to provide context for LLMs to create meaningful tests. This approach allows for the creation of tests that simulate complete user journeys and set the appropriate system state. Honeycomb is utilized as a tool for visualizing and exploring data to refine the internal models used by Traceloop. The combination of OpenTelemetry's extensive data and LLMs offers significant potential for smarter code and test generation as well as the modeling of complex systems.

Earlier this month, Honeycomb released Query Assistant, a natural language querying interface that allows users to express desired queries in plain English, which the system then translates into Honeycomb queries. The development of Query Assistant involved addressing several challenges associated with integrating large language models (LLMs), such as managing context windows, ensuring prompt engineering effectiveness, and dealing with latency issues inherent to commercial LLMs like GPT-3.5-turbo. The team focused on creating a user-friendly product rather than simply wrapping an LLM interface, emphasizing the importance of prompt engineering to accommodate broad and vague inputs while implementing measures to mitigate prompt injection risks and ensuring non-destructive and undoable actions. Additionally, the project required significant legal and compliance considerations, especially for privacy-sensitive customers, and highlighted the inadequacy of early access programs in fully testing the robustness of such features. Ultimately, the effort underscored that LLMs should be seen as engines for features rather than standalone products, requiring careful integration into existing product frameworks.

Development teams face considerable pressure to innovate and release software quickly, often leading to burnout among 83% of software developers. Observability can alleviate this pressure by providing real-time insights into complex systems, enabling rapid issue remediation and improving release velocity and developer morale. At Slack, implementing Honeycomb’s event-based observability reduced flaky tests from 50% to 5%, significantly increasing release speed. Observability, when integrated into development processes, offers real-time feedback on code behavior, empowering teams to identify and fix issues swiftly. Honeycomb’s platform, combined with OpenTelemetry, enhances visibility into distributed systems, ensuring performance and resource utilization are maintained. By aligning error thresholds and creating a supportive culture, observability fosters frequent, smaller deployments, reducing stress and promoting creativity. Honeycomb emphasizes that incorporating observability isn't just a technical enhancement but a cultural shift toward shared problem-solving and sustainable, accelerated development.

Honeycomb introduced a new Query Assistant feature that leverages ChatGPT to generate queries from natural language questions, with tracing built into its development from the outset to ensure smooth operation. The development team, including Tanya Romankova and Craig Atkinson, focused on enhancing visibility into how the feature operates, especially concerning the time taken by different processes. For instance, they discovered that while the API call to ChatGPT took 840ms, additional steps such as constructing the prompt and truncating the list of fields accounted for significant delays, leading them to implement optimizations. Through observability, they improved the feature's performance by refining these processes, demonstrating how such insights can enhance development and maintain visibility in production. The announcement encourages users to explore this feature by signing up for Honeycomb.

Event-based observability, as championed by Honeycomb, offers a transformative approach to debugging complex cloud systems by providing rapid incident response and context-rich data that empower any team member to resolve issues. Unlike traditional monitoring and logging methods, which can be time-consuming and lack detailed context, event-based observability employs distributed tracing to allow teams to quickly identify and address unknown problems by analyzing high-cardinality data in real time. This approach not only speeds up incident response times, saving organizations substantial costs and increasing revenue by improving system uptime, but also fosters better work-life balance for engineers by reducing the need for overtime and unnecessary alerts. Honeycomb's platform leverages machine-assisted debugging tools like BubbleUp and a generative AI-powered Query Assistant, enabling even less experienced engineers to effectively troubleshoot and understand user experiences, ultimately leading to reduced toolchain sprawl and lower operational costs.

Honeycomb's design team embarked on enhancing the Lattice design system, beginning in early 2021, focusing on optimizing typography, color, and spacing while addressing design inconsistencies through thorough audits. By 2023, they established a structured process for component creation and collaboration, incorporating feedback from internal stakeholders such as engineers and designers. The team emphasized documentation, accessibility, and establishing best practices to streamline component contribution and improve the overall design system. Efforts included reorganizing Lattice, aligning Figma components with engineering components, and implementing naming conventions to foster better collaboration between designers and engineers. The introduction of an Asana board and Storybook for component tracking, along with comprehensive usage guidelines, aimed to ensure consistency and clarity. The team received positive feedback on these changes, which motivated further contributions and enhancements, and they emphasized the importance of a flexible process that empowers team members to contribute at various levels. The approach acknowledges that design systems are evolving entities, requiring ongoing user feedback and iterative improvements, encouraging the emergence of new ideas and collaborative action.

Smoke testing, also known as confidence testing, is a preliminary testing method used to identify severe software failures by focusing on critical software aspects. Originating from electronic hardware testing, the term reflects the idea of checking whether "smoke" emerges when "power" is applied. At Honeycomb, smoke tests are implemented for OpenTelemetry distributions by running example applications in a container and analyzing the telemetry exported to an OpenTelemetry Collector using Bash scripts, jq, and the bats-core framework. This process involves configuring SDKs, employing instrumentation libraries, exporting telemetry, and validating the json output against expected results, such as span names and attributes. The methodology offers several advantages, including demonstrating library usage, ensuring error-free builds in clean environments, and confirming successful telemetry exporting and parsing. By saving telemetry data as json files, Honeycomb achieves repeatable and precise testing across projects using a language-agnostic framework, with results uploaded as artifacts in CI workflows. While smoke tests form a crucial part of the testing suite, they are most effective when combined with unit tests and automated checks, providing confidence throughout a project's lifecycle.

Honeycomb Metrics enhances observability platforms by seamlessly integrating system metrics with application observability data, eliminating the need for tool or team switching. To maximize benefits, it is crucial to ask comprehensive questions about system performance, as simply knowing when builds fail isn't enough for diagnosis. Wide events, while useful, don't accommodate every metric, thus requiring adaptation, especially for older systems with custom metrics. The open-source standard OpenTelemetry (OTel) facilitates the transmission of telemetry data across various platforms, allowing even proprietary metrics to be integrated with minimal effort. By mapping metrics to events using timestamps, teams can consolidate data points into single events, reducing event throughput and simplifying customization. Heroku's approach demonstrates how leveraging these strategies can effectively blend metrics into observability platforms, making metrics more than just an ancillary component.

The text explores the comparative strengths and weaknesses of metrics, traces, and logs in application monitoring. It critiques metrics for their lack of context and dimensionality, arguing that they are best suited for infrastructure monitoring rather than understanding user behavior or system intricacies. While metrics are cheaper to store due to their aggregated nature, they require predefined queries, limiting their flexibility. In contrast, traces provide richer context and granularity, allowing for more dynamic analysis of system behavior, although they are more expensive to store. Logs are considered less useful than spans within traces, as they lack contextual information. Ultimately, the text advocates for a trace-centric approach to observability, suggesting that metrics be used when context is absent and encouraging the use of spans over traditional logs for more comprehensive insights.

In April 2023, Honeycomb introduced several new features and improvements to enhance user experience and data management capabilities. Notably, the Query Assistant was launched, enabling users to interact with the Query Builder using natural language, thereby simplifying data querying without prior knowledge of query syntax. This feature is accessible to all users at no extra cost. Additionally, Honeycomb introduced Deployment Protection Rules for GitHub Actions, allowing developers to block deployments based on query results to prevent potential negative impacts on user experience. The platform also released a guide on integrating OpenTelemetry with Honeycomb, highlighting the benefits of application instrumentation and data visualization. Several user interface and experience updates were made, such as resizable trace waterfall sidebars, improved trigger functionality with autocomplete and direct links, and the ability to overlay charts in query results for better data comparison. These enhancements aim to streamline workflows and improve the overall functionality of Honeycomb for its users.

Honeycomb has launched Query Assistant, a new feature designed to make querying more accessible by allowing users to input natural language queries, such as "slow endpoints by status code," to generate relevant queries without needing to master complex query languages. This feature aims to enhance user participation across different teams by simplifying the process of interacting with observability data. While Query Assistant is intended to complement Honeycomb's existing Query Builder UI, it underscores the company's philosophy of using AI to augment human intuition rather than replace it. Despite its current limitations, such as less accuracy with large schemas, Honeycomb is committed to refining Query Assistant based on user feedback and ensuring privacy through robust data protection measures. The company envisions AI as a tool to empower users, focusing on enhancing natural language querying and providing tailored insights, while steering clear of automated alerting and anomaly detection, which they view as less effective applications of AI.

Honeycomb has introduced a groundbreaking feature, Query Assistant, which integrates generative AI to allow engineers to interact with its observability platform through natural language queries, thereby democratizing access to observability tools for users with varying levels of expertise. This innovation eliminates the need for knowledge of complex query languages like SQL, enabling developers to ask questions in plain English and receive modifiable query results, thus enhancing human intuition and facilitating faster feedback on code performance and behavior. By leveraging OpenAI, Honeycomb's new capability revolutionizes traditional observability approaches, offering a unique, intuitive alternative to conventional APM tools that focus on data analytics and automated alerting. Query Assistant, alongside other tools like BubbleUp, aims to streamline the debugging process by enabling quick identification of issues without the cumbersome task of navigating through metrics dashboards, logs, and traces. Honeycomb's approach aligns with its philosophy of making observability user-friendly, further supported by its innovative pricing model and collaborative features, ensuring that understanding application behavior in complex cloud environments becomes an effortless team effort. The Query Assistant is currently available as an experimental feature for all Honeycomb users, with assurances that no user data is shared with OpenAI or retained for model training.

A vibrant collection of artistic data visualizations, dubbed #chArt, has emerged from the Pollinators Slack community, celebrating the fusion of charts and art, often resulting in unique and imaginative creations. This gallery includes various submissions from community members, featuring whimsical interpretations such as a "raincloud" reminiscent of "The Truman Show," a heatmap humorously described as sentient, serene "mountains," abstract art reminiscent of early computer graphics, and even likenesses to pop culture icons like "Porg" and "Predator." Among the highlights are creative renditions of ocean waves, the iconic Haystack Rock, and a striking "Honeycomb" logo within a heatmap that showcases the playful intersection of data visualization and creativity. The community invites enthusiasts to join the conversation about OpenTelemetry and related topics, encouraging participation in future showcases while also offering a chance to create winter-themed heatmaps through the "happy o11ydays" repository.