October 2022 Summaries

Despite not having a dedicated platform team, Honeycomb emphasizes the importance and evolving role of platform engineering within modern organizations, highlighting the distinction between internal platform teams and external vendors. While platform teams differ from DevOps by focusing on higher-level abstractions and vendor management, their role is crucial in bridging the gap between vendor offerings and company-specific needs, ensuring that infrastructure aligns with business goals and market differentiation. Even in a future where cloud services become even more comprehensive and self-service, platform teams will continue to be essential in optimizing costs, evaluating architecture decisions, and enhancing development efficiency without necessarily writing code. The work of platform teams is pivotal, potentially influencing a company's operational success, availability, and development speed, as they navigate an increasingly abstracted and powerful technological landscape.

OpenTelemetry (OTel) is an open standard for distributed tracing and metrics collection, facilitating observability across complex systems. Advocated by Honeycomb, OTel allows seamless integration with multiple backends without necessitating code changes. In a Honeycomb-hosted roundtable with industry professionals, best practices for utilizing OTel were discussed, highlighting the advantages of combining auto-instrumentation with manual methods for comprehensive data collection. Emphasis was placed on using the OpenTelemetry Collector for efficient data handling, ensuring security through data anonymization, and leveraging statistical sampling to manage data volume while preserving significant insights. Honeycomb's Refinery was recommended for trace-aware sampling, enhancing data analysis efficiency. The conversation underscored OTel's flexibility and adaptability in supporting robust observability frameworks.

Honeycomb offers a novel approach to software telemetry by integrating logs, metrics, and traces into a unified, interactive system that challenges traditional, siloed data storage methods. With its innovative Service Map and enhancements to the BubbleUp feature, Honeycomb enables users to explore and understand their systems dynamically, offering tools to analyze outliers and compare subsets of data against baselines. This approach allows engineers of varying experience levels to transition from identifying anomalies to understanding their causes and solutions, fostering a more intuitive and connected exploration of data. The platform emphasizes flexibility and user empowerment, encouraging seamless navigation between visualizations and data queries, ultimately aiming to redefine the standard for observability tools by eliminating investigative dead ends and enhancing the user experience.

Honeycomb has introduced the Service Map, a feature designed to enhance the visualization and understanding of complex service environments by allowing users to analyze and filter communication between their services. This tool is part of Honeycomb Enterprise plans but is temporarily available to all users for free until the end of 2022, providing an opportunity for evaluation before potential upgrades. Users with Honeycomb Classic datasets must migrate to Environments to utilize this feature, which automatically generates maps from trace data. The Service Map offers advanced filtering capabilities, enabling users to gain insights into service performance, relationships, and dependencies, and assists in tasks such as onboarding and instrumentation validation. It also allows users to explore specific traces and performance metrics over customizable time ranges. After 2022, the Service Map will be exclusive to Honeycomb Enterprise users, encouraging those interested to explore its capabilities during the promotional period.

Honeycomb, a leading observability platform, announced the launch of its Service Map and enhancements to the BubbleUp tool, both designed to improve the understanding and debugging of complex cloud applications. These innovations allow developers to quickly visualize and analyze vast amounts of telemetry data to identify and resolve issues more efficiently. The Service Map offers an interactive, visual debugging tool that reveals relationships between services, allowing users to filter and overlay traces to uncover underlying issues. Meanwhile, BubbleUp's enhancements enable users to identify commonalities in problematic data by leveraging machine analysis, thereby accelerating the debugging process without switching between tools. These features, aimed at streamlining the observability process, are available to different tiers of Honeycomb users, with the Service Map becoming exclusive to Enterprise customers after a promotional period.

The Honeycomb Buildevents Action is a tool designed to enhance the performance of build pipelines in GitHub Actions by generating distributed traces that help identify performance bottlenecks and issues such as flaky tests. By utilizing the gha-buildevents Action, users can instrument their workflows to include metadata from the GitHub Actions environment, providing visibility into each step's duration and output details, which can illuminate patterns and inefficiencies across multiple builds. A new hands-on guide offers a comprehensive walkthrough of using this tool, detailing how to set it up with a sample application and configure it for optimal performance analysis using Honeycomb's interface. This guide is particularly useful for those new to GitHub Actions or Honeycomb, as it illustrates practical examples of integrating buildevents into workflows to improve delivery pipeline efficiency. Additionally, the guide requires a free Honeycomb account, a GitHub account, and about 30 minutes to complete the exercises, enabling users to gain a practical understanding of these tools in action.

The release of the Datadog APM Receiver simplifies the integration of existing Datadog traces into Honeycomb by leveraging the OpenTelemetry Collector, providing a seamless way to evaluate Honeycomb without re-instrumenting applications. This innovation allows users to send Datadog traces to any OpenTelemetry-compatible endpoint, facilitating a smoother transition and avoiding vendor lock-in by adopting the widely supported OpenTelemetry standard. The process involves using the OpenTelemetry Collector Builder to create a custom distribution, which can then receive Datadog traces and send them to Honeycomb. This setup enables organizations to assess Honeycomb's suitability while using their current Datadog instrumentation, despite the limitation that the receiver only ingests tracing data. Although this solution is intended as a temporary measure, its implementation highlights the strategic advantage of transitioning to OpenTelemetry, which is endorsed by numerous observability vendors and is the second most popular CNCF project.

Network topology in the cloud can become complex, particularly when transmitting data to external SaaS providers, but AWS PrivateLink offers a streamlined solution for AWS infrastructures. Honeycomb has introduced an AWS PrivateLink endpoint for enterprise customers, enabling telemetry data to remain within the AWS network, thus reducing attack vectors and eliminating the need to configure outbound egress firewalls for sending data. This approach can simplify network topology further by potentially replacing NAT gateways, which can also lead to cost savings due to AWS's data processing charge on traffic through NAT gateways. Additionally, using PrivateLink endpoints with other vendors or AWS services can result in aggregated data transfer discounts, as AWS bills this combined data transfer and provides discounts as the volume of data increases. There is no extra cost for using AWS PrivateLink, and interested enterprise customers are encouraged to consult the documentation and contact their Customer Success representative for setup assistance.

The text provides a detailed guide on deploying and configuring an OpenTelemetry Collector using Helm in a Kubernetes cluster to manage observability data uniformly. It emphasizes the importance of customizing the default configuration to meet specific requirements by iterating changes within a test environment. The process involves installing the OpenTelemetry Collector Helm chart, creating a `values.yaml` file for configuration, and repeatedly updating the installation to refine the setup. The guide also addresses troubleshooting common issues such as pod status problems and port configurations, and suggests looking at logs for further insights. Additionally, it highlights the benefits of using Honeycomb for observability and offers guidance on setting up pipelines and exporters within the Helm chart's configuration.

A new community-contributed Gradle plugin has been developed to optimize and debug Gradle builds using OpenTelemetry, a vendor-neutral observability framework. This plugin, which can be used with a free Honeycomb account, provides a trace view of each task in the build, allowing users to identify bottlenecks and opportunities for improvement. By sending telemetry data to Honeycomb, users can visualize the execution of tasks and test cases, facilitating quick debugging of build issues. Honeycomb’s BubbleUp feature further aids in analyzing performance differences between tasks. The plugin is compatible with Android Gradle builds and is available on the Gradle plugin portal. To start using it, developers need to configure their Gradle build file to send data to Honeycomb, with support for securely storing API keys. The plugin is designed to enhance the visibility and performance of Gradle builds, and all features are accessible through Honeycomb’s free plan.

An outage on September 8th, 2022, caused significant interruptions in the ingest system for over eight hours, revealing critical vulnerabilities in the system's architecture, particularly with the in-memory caching mechanism of Shepherd hosts. Despite initial efforts to stabilize the system through vertical scaling and aggressive sampling, the issue persisted, manifesting as a “shark fin” pattern in system performance graphs, where accumulated requests completed simultaneously but unpredictably. The Engineers discovered that the cache's locking mechanism led to bottlenecks when entries were backfilled, causing cascading failures in the Shepherd and Refinery clusters. A decisive fix involved pre-filling the cache, which successfully improved system stability, although the root cause of the cascading failures remained elusive. The incident highlighted the challenges of managing complex systems under stress, underscoring the importance of accurate mental models, the impact of technical issues on engineering teams, and the necessity for long-term architectural improvements to support growth.

Honeycomb has announced several updates to its OpenTelemetry support, including the general availability of OpenTelemetry logs, which now allows users to send and query logs through various methods, with logs appearing as events on the Home page and displayed as Span Events in the Trace UI. This update facilitates the correlation of logs with traces, improving observability without requiring a complete rewrite of existing systems. Additionally, Honeycomb has released version 1.0 of its OpenTelemetry .NET SDK, featuring easy configuration and enhanced extensibility, including support for JSON with OTLP/HTTP, eliminating the need for translation via a Collector for frontend applications. Looking ahead, Honeycomb plans to develop SDK distributions for Node.js, Python, and Ruby, and is focusing on easing the transition from Beelines to OpenTelemetry while continuing to contribute to upstream OpenTelemetry improvements.

BubbleUp, a feature of Honeycomb, enhances debugging by using machine analysis to identify the commonalities in problematic data compared to baseline data, thereby streamlining the debugging process. Unlike traditional AIOps tools, BubbleUp addresses the challenges of unpredictable system behavior by allowing users to surface issues from various parts of the Honeycomb UI, beyond just heatmaps. The recent enhancements enable users to apply powerful pattern detection to specific user groups or conditions, facilitating a broader range of queries and insights. With features like Result Filtering and BubbleUp Permalinks, users can achieve faster analysis and share results easily for improved collaboration. These improvements make BubbleUp more accessible, allowing both novice and expert users to gain insights into their applications more efficiently, with access to all functionalities available in Honeycomb's free plan.

In the Authors’ Cut series, the discussion revolves around observability within the context of significant technical and cultural shifts like cloud-native, DevOps, and Site Reliability Engineering (SRE). Observability is crucial for managing the complexity inherent in modern systems, allowing platform developers to embed observability practices from the start and continuously evolve them. The narrative highlights the importance of both the technical and human aspects of observability—emphasizing that understanding and communication among team members are as vital as the data itself. Through examples like Honeycomb's chaotic engineering experiment and incident reviews, the text illustrates how observability enables effective debugging and analysis, shifting the focus from isolated data interpretation to collaborative problem-solving. It underscores the dynamic nature of cloud-native environments, where observability plays a key role in sustaining and improving system reliability and team operations, ultimately enhancing practices like feature flagging and progressive release patterns. The series concludes with a nod to future content and an invitation to explore Honeycomb’s offerings in modern observability.

Liz Fong-Jones, who joined Honeycomb nearly four years ago after a successful career at Google, has been appointed as the company's Field CTO. Her journey at Honeycomb began as the first developer advocate and principal engineer, a risky decision given the company's startup status and financial uncertainties at the time. Despite the challenges, Liz has become a pivotal figure in the company, spearheading major initiatives like developing Service Level Objectives, adopting OpenTelemetry, and reducing compute costs with Graviton, while also contributing to fundraising and co-authoring "Observability Engineering." Her appointment reflects Honeycomb's values of engineering excellence, autonomy, and mutual respect, as well as her advocacy for structural representation and dignity in the workplace. As Field CTO, Liz will focus on working with executive teams at strategic and prospective customers, continuing to drive innovation and growth within the company.

Honeycomb emphasizes that the transition to observability in data systems requires both technical and sociological changes, focusing on how organizations can solve cultural challenges and demonstrate business value. It suggests addressing the frequent paging of staff by adopting service-level objectives (SLOs) to reduce noise and improve focus on critical issues, which requires leadership buy-in and side-by-side experimentation to avoid risking current alert management systems. The text also advocates for the adoption of OpenTelemetry and distributed tracing to enhance debugging efficiency and avoid vendor lock-in, arguing that this approach reduces cognitive load and facilitates quicker issue resolution. It stresses measuring improvements in execution capabilities rather than just the number of bugs, highlighting the importance of metrics like time to recover from failures and time between commits and production. The overall message encourages starting with small steps in observability and leveraging frameworks like OpenTelemetry to align with business outcomes, making a case for continuous delivery and agile practices as foundational elements in this transition.

A new community-contributed integration developed by a Honeycomb employee connects Honeycomb's observability tools with ServiceNow's ITSM systems, aiming to improve the historically weak integrations between these platforms. This integration allows users to create or update configuration items in ServiceNow's CMDB, generate rich Service Maps based on real traffic data, and observe the impacts of change requests directly in Honeycomb. It also facilitates the creation of incidents or events in ServiceNow for Honeycomb Triggers and SLOs, enabling streamlined and efficient workflows. The integration, which requires a Honeycomb Enterprise subscription and a ServiceNow Flow Designer subscription, is designed for easy setup without the need for additional agents or subscriptions. The developer encourages feedback from the community to enhance its utility and invites users to try out the integration and provide input on its functionality.

In September, Honeycomb introduced several updates to enhance its API functionalities, data integrations, and tooling. Notably, the dataset API now allows programmatic updates to dataset definitions, and the SLO API includes new features for tracking compliance and budget details. The platform has expanded its support for OTLP data ingestion, now including HTTP/JSON, and released improvements to its Refinery trace sampling tool, emphasizing better validation and diagnostics. A new integration with Terraformer simplifies managing Honeycomb resources as code, while the PagerDuty integration now attaches relevant graph images to alerts. GitHub Actions users benefit from the updated gha-buildevents utility, which offers comprehensive workflow tracing capabilities. Additional updates include enhancements to the Derived Column editor and the resolution of a copy button issue in documentation, with all changes documented in Honeycomb’s changelog and shared on social media platforms.