September 2024 Summaries

Engineers at Honeycomb found themselves facing a persistent issue with outdated Slack aliases that complicated incident management during on-call rotations. This problem arose because some teams shared pager rotations without hand-off meetings, causing disruptions in communication when incident management tools like Jeli and PagerDuty failed to invite the correct individuals to incident channels. Despite the existence of potential solutions, many were complex and costly, requiring infrastructure that Honeycomb preferred to avoid. Instead, a more straightforward approach was devised by creating a script that synchronized on-call schedules using PagerDuty and Slack APIs, which was then automated to run hourly via a Kubernetes cronjob. This small yet impactful solution was met with enthusiasm from the engineering team, highlighting the importance of addressing even minor operational frictions. The success of this initiative prompted the decision to open-source the script, allowing other organizations facing similar challenges to benefit from the streamlined process without incurring additional costs.

In a blog post update, the author revisits the OpenTelemetry Transformation Language (OTTL) to demonstrate how it can streamline telemetry transformation within the OpenTelemetry Collector, especially with the v0.120.0 release enhancing usability. The post provides a series of practical examples or "recipes" that showcase various real-world scenarios where OTTL effectively handles complex data transformations, such as setting attributes based on conditions, replacing characters in attribute keys, parsing JSON bodies into attributes, and managing resource attributes. It highlights the flexibility and power of the Transform processor to manipulate data in-flight, underlining its capability to meet diverse requirements by dynamically adjusting telemetry as it moves toward its destination. The post also emphasizes the synergy between OpenTelemetry and Honeycomb for observability, suggesting additional resources for those interested in delving deeper into the advantages of combining these technologies.

Honeycomb has launched Honeycomb for Frontend Observability, a new tool that extends its debugging capabilities to the frontend of web applications, offering engineering teams comprehensive insights into performance and user experiences. This tool addresses the complexity of modern interactive sites by overcoming the limitations of traditional frontend monitoring tools like real user monitoring (RUM), which fail to provide real-time, detailed context for user interactions. Honeycomb’s solution uses OpenTelemetry-based instrumentation to allow for seamless integration of custom attributes, providing powerful visualizations that reveal new insights into customer and performance issues. This platform has been positively received by early adopters, such as Fender Instruments, for its ability to accurately track and diagnose site performance issues, enhancing user experience and optimizing conversions. The new offering integrates with Honeycomb’s existing platform, allowing for a unified debugging interface across the entire application stack and is available now for enterprise customers.

Honeycomb for Frontend Observability is a newly available tool designed to enhance developers' ability to debug and optimize their web applications by providing rich observability data and real-time insights into user interactions. Unlike traditional Real User Monitoring (RUM) tools, which often require piecing together various outputs to identify issues, Honeycomb offers a comprehensive solution with its open-source instrumentation package and Web Launchpad, capable of displaying end-to-end traces and high-cardinality data without pre-aggregation. This allows engineers to efficiently identify root causes of poor Core Web Vitals (CWVs) scores by leveraging thousands of custom attributes and integrated anomaly detection. A case study with Iron/Out, an ecommerce performance optimization firm, highlights the tool's effectiveness in streamlining CWV debugging and continuous monitoring, enabling swift identification of performance bottlenecks, such as issues related to the Interaction to Next Paint (INP) metric. By providing easy access to detailed performance metrics and enabling deep dives into user behavior, Honeycomb empowers teams to proactively address issues, consolidate observability tooling, and improve the overall user experience.

Interaction to Next Paint (INP) is the latest addition to Google's Core Web Vitals, serving as a metric to gauge the responsiveness of a web page by measuring the time between a user interaction and the browser's ability to render the next frame. This metric, alongside Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS), aims to quantify crucial aspects of user experience online. Honeycomb for Frontend Observability offers tools to debug and optimize INP scores by capturing detailed attribution data for each event, allowing developers to analyze and address issues in web performance. Honeycomb's BubbleUp and fast querying functions enable users to explore data, identify problematic areas, and propose optimizations, although the INP metric currently only reports data from Chromium-based browsers, potentially omitting significant portions of user interactions on iOS devices using Safari. By understanding and improving INP scores, developers can enhance the overall user experience on their websites.

Modern DevOps and SRE teams have evolved their logging and monitoring strategies to improve system integrity and performance in complex IT environments by adopting best practices that streamline troubleshooting, enhance performance monitoring, and bolster security. Effective logging provides clear insights into system behavior, facilitates the identification of bottlenecks and anomalies, and aids in detecting potential threats. Key practices include using structured logs, consolidating log entries, employing unique identifiers, standardizing log fields, avoiding the logging of sensitive data, and utilizing a centralized logging management system. These practices ensure that logging remains efficient, actionable, and scalable, benefiting from real-time alerts and proper documentation to guide developers and stakeholders. Tools like Honeycomb support these practices by offering advanced observability and log analysis capabilities, allowing for the extraction of valuable insights from vast amounts of log data, thus enabling better understanding and improved system performance.

Building a Center of Production Excellence (CoPE) involves shifting focus from pre-production environments to production environments, emphasizing the importance of treating production as a critical source of information rather than just a final stage for code deployment. The article argues that while pre-production environments offer a controlled space for testing, they lack the real-world complexities and user interactions present in production, which can result in differing code behavior. By integrating tools like Honeycomb for monitoring and adding Honeymarkers for better traceability of code changes, organizations can enhance their understanding of how code performs in production. This approach encourages the sharing of insights across teams and the organization, fostering cross-functional collaboration and breaking down silos. The text suggests that starting with small iterative changes and promoting engagement with production can lead to improved systems that better serve users and align with organizational goals. Ultimately, it calls for organizational leadership to support CoPE initiatives and highlights the continuous need for intervention and modulation to maintain a dynamic production environment.

Events are a versatile and comprehensive approach to understanding software systems, serving as a foundational concept for platforms like Honeycomb. They function as structured logs, capturing the context or state of a system at a specific time, and can also encompass metrics, which are numeric representations of physical properties. Furthermore, events can act as spans, contributing to the creation of traces that model the flow of functions within a system. This event-based model challenges traditional metrics-based approaches by offering a more effective and elegant solution, akin to the revolutionary shift from a geocentric to a heliocentric model in astronomy. The text argues for the advantages of adopting an event-based methodology, which includes the potential for more accurate insights and reduced computational overhead, while highlighting the importance of being open to new perspectives and letting go of outdated assumptions to address underlying problems effectively.

OpenTelemetry is a comprehensive observability framework that supports application tracing, metrics, and logging across 12 programming languages, providing stable and mature solutions for end-to-end service tracking. The project includes a range of features, such as the OpenTelemetry Collector, which supports various log and metric sources and offers telemetry processing capabilities. While core components are stable, some areas like semantic conventions, client instrumentation, and profiling remain in development, with advancements expected soon. OpenTelemetry is also working on new initiatives, such as control planes, telemetry schemas, and the use of eBPF for black box telemetry, which promise to enhance its functionality and ease of use. The project continues to evolve, focusing on improving performance, reliability, and the user experience, with a vibrant community actively contributing to its growth and development.

The text discusses the evolution of observability in software systems, tracing its roots from the traditional metrics, logs, and traces model (observability 1.0) to a more advanced approach (observability 2.0) that uses wide, structured log events as a single source of truth. This shift is likened to the transformative impact of virtualization on the tech industry, suggesting that observability 2.0 offers greater precision and a more integrated development and operational experience. The traditional observability tools, reliant on metrics, are critiqued for their high costs and limitations in handling complex systems due to their reliance on pre-aggregated data. In contrast, observability 2.0 is presented as a more cost-effective, flexible, and holistic approach, enabling real-time insights and allowing teams to respond more dynamically to changes and issues. The narrative emphasizes the importance of adapting to this new model to stay relevant in the rapidly evolving landscape of software development and encourages innovation beyond metrics-based tools to address modern software challenges.

Log management is a crucial practice in the digital age, involving the efficient handling of log data from various systems, applications, and devices to extract valuable insights. This process includes collecting logs, formatting them into a standard format, storing them for both real-time analysis and archiving, and monitoring them for potential issues or security threats. The analysis of logs can help identify performance trends and security risks, while reporting offers detailed insights into system operations. Effective log management supports better observability, improved security, compliance with regulations, and a unified data storage approach. Tools like Honeycomb enhance log management by providing real-time insights, advanced analytics, high-context alerting, and scalability, thereby enabling organizations to quickly detect and respond to issues and understand their software's usage and operations comprehensively.

Engineering has evolved significantly from delivering discrete, point-in-time products to embracing a continuous development and service delivery model driven by cloud computing and SaaS. This transformation has shifted the engineering focus from overcoming technical challenges to prioritizing team coordination and collaboration, as emphasized by Nick Travaglini of Honeycomb. The feedback loop between engineers and users is now faster, enabling real-time improvements in software performance, with observability 2.0 playing a crucial role by centering on the customer's experience rather than just system health. Observability 2.0 provides engineers with a unified source of truth for metrics like latency and error rates, allowing them to prioritize improvements based on real user needs. As businesses become more service-oriented, tools like AI and advanced platforms are freeing engineers to concentrate on customer-centric innovations. Josh Clayton from HappyCo highlights that modern engineering is about understanding and enhancing the user experience, with observability 2.0 being central to this evolution by equipping engineers with the insights needed to align with customer expectations and deliver exceptional value.

Sampling is essential for managing large-scale applications, and Honeycomb recommends using their Refinery tool to effectively sample data. The rules file provided is an example template for setting up Refinery, consisting of seven rules designed to minimize event volume by dropping uninteresting data while retaining rare or significant data such as errors and anomalies. The rules employ three types of samplers: the Rules-Based Sampler, the EMA Dynamic Sampler, and the EMA Throughput Sampler, which sequentially assess traces to determine the appropriate sampling action. The strategy begins with specific rules for keeping and dropping data, proceeds with dynamic sampling for frequently occurring events, and concludes with a catch-all rule to manage remaining traces. The philosophy underpinning these rules is to discard routine data like fast, successful requests and health checks, while preserving data that may indicate issues or require attention. Users are encouraged to regularly review and adjust their sampling rules to accommodate new services and changing data patterns, ensuring optimal sampling performance as their infrastructure evolves.