May 2024 Summaries

A Center of Production Excellence (CoPE) plays a crucial role in cloud-native software organizations by ensuring safe, reliable, and sustainable systems through effective practices and tools. The CoPE's independence is vital as it allows autonomy in perceiving and affecting change without interference from existing power dynamics within the organization, which may resist alterations due to vested interests. Independence is complemented by the CoPE's involvement in work processes, ensuring it does not merely dictate changes but actively participates and understands the challenges faced by teams. Additionally, being informed and informative allows the CoPE to study actual work behaviors and communicate insights to stakeholders, acting as an advisor that facilitates smooth collaboration and decision-making. The CoPE must be insulated enough to deliver candid assessments, even when such insights are unwelcome, to foster adaptability and effective organizational change.

Honeycomb's storage engine, known as Retriever, has undergone internal changes to enhance performance while maintaining the same user experience. Retriever operates through two processes: a writer that appends event data to disk, and a reader that processes queries by calculating aggregations from the stored data. With the adoption of a new data model for environments and services, Retriever was adapted to support multi-dataset queries, allowing for more efficient querying across multiple services. This led to the development of "virtual datasets," which map datasets to container datasets, enabling more efficient queries by reducing the need to read irrelevant data. This change improved query runtimes significantly, especially for complex environments with numerous services, reducing median query duration from around 20 seconds to 0.2 seconds. The new system also provides the flexibility to add features without impacting query performance, and Honeycomb plans to continue leveraging this model to further improve query efficiency.

Honeycomb has expanded its support business hours to accommodate European customers, offering service from 9:00 a.m. to 5:00 p.m. GMT/BST starting May 20th, 2024, in addition to its existing service hours in the U.S. This change aims to enhance customer experience without affecting the pricing or level of technical support, which remains consistent across all regions and plans. Customers can reach out to Honeycomb Support through various channels, including the support portal, in-app options, and email, to address issues, ask questions, and provide feedback. The company also offers resources such as technical documentation, educational courses, a community Slack channel, and scheduled office hours with developer advocates to further assist users in optimizing their observability stack. Additionally, Honeycomb encourages those interested in European data residency to contact their sales team, with a self-serve option anticipated in the future.

Software systems often operate in a "degraded mode," requiring a mix of technical, organizational, and human effort to maintain stability, with some organizations leveraging resilience as an active practice rather than an inherent trait. The concept of a Center of Production Excellence (CoPE) or an Observability Guild is discussed as a means to enhance organizational resilience by creating a dedicated group that can critically analyze and improve internal processes. This group should have a degree of authority and autonomy, drawing parallels to safety departments in organizations, and requires diverse, experienced members from across the organization to effectively evaluate and recommend changes. The CoPE employs both passive and active tactics to improve observability practices, such as regular training, newsletters, and incident response planning, thereby fostering a culture of continual learning and adaptability. Top-down support from management is crucial, involving autonomy for the CoPE, empowering frontline expertise, and encouraging diverse perspectives to prevent knowledge silos. By nurturing adaptability and production excellence, organizations can better respond to the dynamic nature of software systems and enhance their overall resilience.

Charity Majors' whitepaper, "The Cost Crisis in Metrics Tooling," delves into the escalating costs and diminishing returns of metrics-based observability tools, which are becoming increasingly detached from the value they provide. The discussion highlights the complexities of managing costs associated with metrics, especially when considering the high volume of custom metrics generated by factors such as the number of hosts, endpoints, status codes, and aggregation methods. The paper argues that while metrics are a mature and widely used technology for monitoring systems, their reliance on time-series databases limits their contextual value, leading to unpredictable and often exorbitant costs. Majors suggests that a shift towards relational data storage is necessary to align costs more closely with value, as current practices can lead to significant expense increases without corresponding improvements in system understanding or debugging capabilities. The full whitepaper further explores how experienced teams manage these costs and the potential of observability 2.0 models, advocating for a more structured approach to logging and metrics usage.

Over the past five years, the complexity of software systems has increased, making it difficult for organizations to maintain reliability while innovating in a challenging macroeconomic environment. Understanding the distinctions between observability, telemetry, and monitoring can help organizations manage these complexities. Observability is a continuous process of analysis using telemetry data, which includes logs, metrics, and traces to understand software systems, while monitoring involves the active or passive observation of these signals to ensure system performance. The rise of modern observability practices, enhanced by tools like OpenTelemetry and platforms such as Honeycomb, allows for a holistic view of system performance and business impact, enabling proactive problem-solving and strategic improvements. Honeycomb, which positions itself as a leader in the observability field, emphasizes the importance of a strategic approach to observability, moving beyond traditional methods that often result in silos and uncontrolled costs. By adopting a comprehensive observability strategy, organizations can achieve better alignment across engineering and business operations, ultimately enhancing customer satisfaction.

Honeycomb has released a Web Instrumentation package based on OpenTelemetry's browser JavaScript packages, designed to simplify the process of instrumenting web services while providing richer insights. The OpenTelemetry auto-instrumentation package offers significant data akin to proprietary real user monitoring (RUM) agents but is open-source, reducing vendor lock-in concerns. Honeycomb's distribution enhances this by incorporating additional data collection, such as Core Web Vitals and session context, helping users understand the causes behind performance metrics and improve web optimization efforts. The package integrates seamlessly with Honeycomb and other vendors accepting OpenTelemetry, offering a streamlined setup and comprehensive insights into document load and user interaction data. Honeycomb's enhancements allow for deeper inquiries into web service performance and user experience, providing actionable insights into web optimization challenges.

Relational fields have been introduced as a new feature that enhances querying capabilities by allowing users to analyze spans based on their relationships within a trace, such as root, parent, or any single span. This advancement enables more complex queries that were previously challenging, such as identifying errors in specific parts of a process, optimizing performance without needing exact span names, and improving customer support by efficiently tracing issues through relational attributes. By using relational fields, developers can now perform comprehensive analyses without needing to propagate specific values like user IDs across all spans, significantly streamlining error detection and performance optimization in complex workflows.

Honeycomb has launched an early access program for its new tool, Honeycomb for Frontend Observability, designed to help teams optimize their web applications by providing in-depth analysis of Core Web Vitals through enhanced OpenTelemetry instrumentation. Available as an NPM package, this tool allows developers to collect detailed attribution data on web performance metrics, enabling them to identify and address issues more effectively. The tool integrates seamlessly with Honeycomb’s existing data analysis features like BubbleUp, Service Map, and SLOs, offering a comprehensive suite to improve performance and customer satisfaction. Unlike traditional real user monitoring (RUM) solutions, which struggle with high-cardinality data and are limited in their ability to drill down into customer journey touchpoints, Honeycomb for Frontend Observability provides extensive contextual data and full stack traces, facilitating faster issue resolution and improved collaboration across teams. The open-source Honeycomb Web Instrumentation package is accessible to all, while the early access program for the Web Launchpad view is available to Honeycomb Enterprise plan customers, promising a more robust approach to debugging and performance enhancement in modern web applications.