June 2026 Summaries

O11yCon highlighted the significant impact AI is having on engineering teams, with companies like Mixpanel experiencing a 50% increase in pull requests, thereby testing their observability infrastructure's capacity. Observability is evolving from a "nice to have" to a necessity, as faster code deployment requires immediate feedback on system performance. Companies like Gem and Bubble are leveraging tools like the Honeycomb MCP server to enhance their incident response and data analysis capabilities, enabling engineers to gain insights without relying solely on experienced personnel. While current AI-assisted observability practices are predominantly reactive, there is a shift towards proactive monitoring, with teams standardizing on OpenTelemetry to provide comprehensive, high-quality telemetry data. This transition is essential for AI to predict issues before they trigger alerts, emphasizing that robust telemetry infrastructure is crucial for effective observability and AI integration in engineering workflows.

AI agents, characterized by their nondeterministic, multi-step, and opaque nature, require comprehensive telemetry to be effectively debugged in production environments. The OpenTelemetry GenAI Semantic Conventions offer a vendor-neutral way to capture necessary attributes, allowing tools like Honeycomb's Agent Timeline to render a complete conversation, including model calls, tool calls, handoffs, and downstream activities, all identified by a shared conversation ID. Proper instrumentation involves threading this conversation ID through the entire execution chain, ensuring that all spans, including those from downstream systems, are connected. This integration facilitates a detailed view of the conversation, highlighting failures and enabling root-cause analysis beyond just the LLM perspective. Instrumentation can be done manually or through auto-instrumentation packages, with the latter providing LLM-layer telemetry while still requiring user-defined attributes for agent and conversation layers. The result is an actionable Agent Timeline that provides insights into conversation dynamics and potential failure points, emphasizing that the root cause of issues often lies beyond the model itself.

Honeycomb recently conducted a rare, scheduled maintenance to upgrade its Kafka cluster infrastructure, resulting in temporary service disruptions in the EU and US regions. This maintenance, which was the first of its kind in five years, aimed to replace aging infrastructure to improve scalability and reliability. During the process, data ingestion continued, but queries and dashboards experienced gaps, causing unexpected customer impact due to insufficient communication and preparation. The company acknowledged the shortcomings in customer notification and impact description, as well as challenges in scheduling and coordinating the maintenance. To address these issues, Honeycomb is implementing new protocols for better communication, extended timelines for internal coordination, and clearer accountability, despite not anticipating similar maintenance in the near future. This experience highlighted the need for robust processes and communication strategies to better manage potential disruptions and maintain customer trust.

Agent Timeline, now generally available to all Honeycomb customers, is designed to streamline the debugging process of agentic applications by reducing mean time to recovery (MTTR) from 40 minutes to five. It addresses the complexities of agentic workflows, which involve multiple agent interactions and tool calls, by providing a comprehensive view of an agent conversation—encompassing agent executions, model calls, tool invocations, handoffs, and retries. This approach contrasts with traditional observability platforms that rely on tracing single threads, which do not capture the full scope of an agent's interactions. Agent Timeline uses high-cardinality, event-based telemetry to present a clear visual representation of agent activities, enabling users to identify issues without switching tools. The platform supports existing OpenTelemetry instrumentation, ensuring compatibility with current setups and allowing users to handle sensitive data securely. Honeycomb's new feature facilitates a deeper investigation into agent behavior across a platform, offering an intuitive way to manage and debug complex agent workflows efficiently.

The second edition of "Observability Engineering" is now available for download, sponsored by Honeycomb, with the physical version expected in a month. Written amidst the excitement and challenges of generative AI advancements between 2025 and 2026, the authors reflect on a shifting industry landscape where AI is acknowledged as not magical but an engineering challenge requiring rigor and new constraints. The book, developed in collaboration with O'Reilly, serves as a timely resource, encouraging readers to revisit foundational principles of observability and adapt to ongoing changes. This edition, enriched by contributions from guest authors, aims to provide valuable insights and discipline in understanding observability, echoing William Zinsser's sentiment that writing is a form of structured thinking.

Honeycomb.io has announced the release of the second edition of "Observability Engineering: Achieving Production Excellence," authored by Charity Majors, Liz Fong-Jones, and George Miranda with Austin Parker, and published in collaboration with O'Reilly. This edition, nearly twice the length of its predecessor, has been rewritten to address the evolving challenges of modern engineering teams, particularly in the context of AI-accelerated development. The book emphasizes the importance of observability in maintaining fast feedback loops in software engineering, underscoring the need for enhanced production telemetry as AI increasingly contributes to code generation. It provides insights into instrumentation for AI-assisted development, debugging LLM-powered applications, and managing telemetry pipelines. The publication coincides with Honeycomb's recent product developments, such as the launch of Agent Timeline and a redesigned Canvas investigation workspace. Featuring contributions from practitioners at the forefront of observability work, the book is available through O'Reilly Media in both print and digital formats.

Observability, once focused primarily on metrics like uptime and error rates, now requires a broader approach as systems become more complex and influenced by AI, demanding a shift from mere operational metrics to a focus on the value delivered to customers and the business. Traditional metrics like MTTR and engineering productivity, while quantifiable, fail to capture the full impact of system behavior on user experience and business outcomes, necessitating a deeper understanding of system performance and its consequences. The evolving landscape necessitates a value-oriented observability strategy that directly links technical performance to product success and customer satisfaction, engaging diverse stakeholders such as product managers, security professionals, and financial officers. AI has amplified this need, highlighting the importance of understanding system behavior beyond availability metrics to justify investments and demonstrate business impact. Organizations must now measure outcomes that matter, like customer experience improvements, product success, and security, to tell a comprehensive value story that aligns with business goals and stakeholder expectations.

In the fourth installment of the Graviton retrospective series, Honeycomb conducted a detailed analysis comparing the AWS Graviton4 and Graviton5 processors by running identical workloads on both in a Kubernetes namespace. Over a 60-day period, it was observed that Graviton5 demonstrated significant CPU efficiency, using 11-26% less CPU for the same workload, and improved performance with a 28% reduction in ingest P99 latency and a halving of the metrics ingest pipeline P99. Tail-based sampling queues were notably shorter by 44-73% at P95, without any regressions across the fleet. The evaluation was conducted using a read-only Canvas to facilitate public interaction with the data. Key services like shepherd, refinery, beagle, newf, and kelpie showed marked improvements, with CPU efficiency gains indicating potential for further cost savings through capacity tuning. This study highlights the substantial performance advantages and opportunities for cost reduction when migrating to Graviton5, making it a compelling choice for those using previous Graviton generations or considering a shift from x86 architectures.

The discussion around token usage in AI and productivity metrics, such as tokenmaxxing, highlights the challenges of quantifying and incentivizing developer productivity through simplistic measures. Drawing on Goodhart's and Campbell's laws, the text argues that relying on quantitative metrics can lead to distorted outcomes and suggests that productivity is a complex, systems-level issue rather than one that can be captured through individual scorekeeping. While tokenmaxxing, likened to the infinite monkey theorem, proposes generating vast amounts of output for potential innovation gains, it also risks creating inefficiencies and unmanageable outputs. The inherent difficulty in measuring productivity parallels challenges seen in previous attempts to quantify work, such as the use of story points. The text stresses the importance of maintaining fundamental team dynamics like psychological safety and robust CI/CD processes, while leveraging AI to enhance these systems rather than simply mandating its use. Ultimately, understanding team dynamics and the obstacles to productivity can provide more valuable insights than focusing on AI-driven outputs alone.

Deploying the OpenTelemetry Collector as an AWS Lambda function can be a viable solution for environments with infrequent telemetry needs, using a Lambda container image and Function URL to handle OTLP/HTTP requests. This setup is cost-effective for low traffic, with cold starts taking about 4 seconds and warm invocations 2–4 ms, but it is unsuitable for high-throughput scenarios due to Lambda's overhead and limitations on retry capabilities. The architecture involves packaging the collector with AWS Lambda Web Adapter to facilitate invocation handling, and it's essential to configure the system to be stateless per request, ensuring no batch processor or sending queue is enabled, as these can hold spans in memory and cause data loss during Lambda freezes. Authentication can be managed using bearer tokens to secure the publicly accessible Function URL, though AWS IAM authentication is not feasible due to the lack of Sigv4 support in OpenTelemetry SDKs. Deployment requires creating an ECR repository and a Lambda execution role, and verification involves checking proper request handling and data delivery to the backend, with troubleshooting focusing on common pitfalls like permission issues and configuration errors.

Working at an observability company doesn't guarantee ease in navigating data, as demonstrated by the challenges faced by the support team at Honeycomb. Previously, the support team grappled with inconsistent dataset naming and the "blank page problem" when trying to build queries to investigate customer issues. Canvas transformed this process by allowing users to describe what they're looking for, automatically building and running queries, and providing visualizations to refine. This shift has enabled the support team to focus on investigation rather than query syntax. Furthermore, the Model Context Protocol (MCP) enhances research by integrating various tools into a single workflow, enabling efficient investigation and minimizing the need for escalations to engineering. The integration of these tools simplifies debugging and alerts investigation, benefiting anyone working in unfamiliar systems.