April 2023 Summaries

The text explores the current state of OpenTelemetry support for React web clients as of April 2023, highlighting the differences in observability between frontend and backend systems. While distributed traces with OpenTelemetry have become a standard for backend systems, their implementation in frontend applications, particularly React web clients, presents unique challenges. The post discusses various aspects of automatic instrumentation in React, such as Document Load and Fetch, and how they track events and user interactions, albeit with verbosity and limitations. It also examines the challenges of connecting user interactions with backend requests and the need for a more cohesive tracing system that reflects the intricacies of frontend operations. The text emphasizes the potential improvements needed in frontend tracing and invites readers to contribute to ongoing efforts within the OpenTelemetry community to enhance client-side observability.

The text discusses the challenges and considerations involved in deciding which incidents warrant a retrospective analysis. It explores the balance between the desire to learn from every incident and the practical limitations of time, resources, and organizational tolerance. Through personal experiences and expert insights, the author highlights the need to selectively conduct retrospectives based on factors such as the incident's learning potential, the risk of exhausting organizational goodwill, and the possible legal or reputational ramifications. The narrative emphasizes that while learning from incidents is crucial, not all incidents justify the effort required for detailed analysis, and certain high-profile or legally sensitive cases may require additional caution. The text underscores the importance of discerning which incidents to focus on to maintain credibility and maximize the learning benefits for the organization.

Determining who should be on call for alerts in systems owned by different teams is a complex and socio-technical challenge, as effective Service Level Objectives (SLOs) often measure interactions between multiple services. The blog post explores various strategies for handling alerting, including not paging anyone at night, paging the on-call engineer for every alert, having a designated team investigate initially before involving others, and using a "switchboard" pattern with a rotation of Site Reliability Engineers (SREs) to manage alerts. Automation can assist in routing alerts by synthesizing signals, but careful definition of fallbacks is essential as novel failures may still require human intervention. The post emphasizes the importance of aligning SLOs with user experience to ensure alerts are meaningful and reflects on Honeycomb's internal practices, such as using E2E monitoring to ensure data accessibility, illustrating the application of the switchboard pattern to address complex service interactions. While there is no one-size-fits-all solution, understanding team dynamics and organizational culture is crucial to developing an effective alerting strategy, and Honeycomb's approach is presented as a potential model to enhance on-call engineer experiences.

Honeycomb has introduced a new Deployment Protection Rule to enhance the GitHub Actions Deployment Protection Rules, providing real-time performance feedback that enables engineering teams to deploy code more confidently and frequently. This new feature leverages Honeycomb's observability capabilities to integrate performance data into CI/CD pipelines, allowing developers to set specific thresholds for deployment decisions based on real-time query results. The rule aims to improve governance in continuous deployment processes by offering an additional layer of protection against performance issues that might not be detected during pre-deployment checks. This collaboration between Honeycomb and GitHub allows for automatic gating mechanisms in GitHub Actions workflows, helping teams catch potential real-time performance issues before they impact production, ultimately enhancing the efficiency and reliability of software releases.

GitHub has announced the public beta of Deployment Protection Rules for GitHub Actions, specifically available to GitHub Enterprise users, which now allows any GitHub App to automatically make decisions in deployment workflows. In collaboration with GitHub, Honeycomb has introduced a Deployment Protection Rule via a GitHub App, enabling real-time performance feedback by running Honeycomb queries before code deployment decisions are made. This addition acts as a pre-deployment gate that uses staging or canary data to prevent deployments that might catastrophically affect production, complementing existing CI checks. Developers can define query payloads in a .honeycomb.yml file within their repositories, specifying conditions under which deployments should be blocked, such as service health measures or unexpected errors. Honeycomb's tool is designed to make deployments more predictable by catching issues that could otherwise escalate post-deployment, while also allowing flexibility in querying various performance indicators. The new feature is available for all Honeycomb users, and GitHub's new automated gating mechanism is intended to increase the safety and efficiency of deployment processes.

Refinery, Honeycomb's tail-based dynamic sampling proxy, is designed to retain interesting traffic while discarding repetitive, less significant data, but its advanced functionality can initially seem counterintuitive to users. Honeycomb's Customer Architect team often fields queries from users struggling to achieve desired sample rates with Refinery, prompting detailed guidance on effective sampling and troubleshooting. Users are advised to ensure diagnostic configurations are correctly set, including using fields like RefineryTelemetry and Debugging options, and to leverage Usage Mode for enhanced visualization of sample rate data. The blog post also emphasizes the importance of tuning the EMADynamicSampler, particularly the AdjustmentInterval, to maintain stability in sample rates and suggests using root. prefixes to limit cardinality. For users seeking greater predictability, the RulesBasedSampler is recommended, offering deterministic tail-based sampling with specific rules for managing traffic. The blog concludes by highlighting the importance of periodically revisiting and refining sampling rules to align with evolving software and business needs, inviting new users to explore Honeycomb's services.

Phillip introduces the latest update on OpenTelemetry (OTel) just in time for Kubecon EU, highlighting the release of a Python SDK distribution designed to simplify data transmission to Honeycomb with minimal configuration and offering features like multi-span attributes, local trace visualization, and deterministic sampling. To get started, users can easily install the necessary packages, instrumentation libraries, and configure environment variables. Additionally, the OpenTelemetry Operator for Kubernetes is emphasized as a reliable and easy-to-use tool that facilitates the installation of an OpenTelemetry Collector across clusters and injects automatic instrumentation into services, addressing the scalability challenge faced by users with numerous services. An exciting development for Go developers involves the introduction of automatic instrumentation using eBPF to extract library data, with support for integrating this into Kubernetes services via the OpenTelemetry Operator. Attendees of Kubecon EU in Amsterdam are invited to visit Honeycomb's booth to learn more about leveraging Honeycomb and OpenTelemetry for enhanced observability in Kubernetes clusters, with demonstrations available on using Helm for quick data ingestion setup.

OpenTelemetry in .NET does lead to performance degradation, but the impact is negligible, particularly in the context of standard .NET applications. The article addresses common concerns about OpenTelemetry's effect on performance and provides evidence to show that the overhead introduced by instrumenting with Activity, adding attributes, and using OpenTelemetry SDKs is minimal, often measured in nanoseconds. The tests conducted demonstrate that while there is a slight increase in latency, such as 16 nanoseconds for adding an Activity and 470 nanoseconds when a TracerProvider listens to the response, these increments are trivial compared to the benefits of enhanced observability and debugging capabilities. Additionally, the use of custom processors in OpenTelemetry can introduce some performance impact, but with careful implementation, these effects can be mitigated. The overall conclusion is that the advantages of using OpenTelemetry for better context during errors outweigh the minor performance costs.

Honeycomb has introduced several updates and enhancements at the end of March, focusing on improving documentation, refining tools, and expanding integration capabilities. The documentation now includes clearer organization and expanded use cases, notably for the OpenTelemetry Collector Deployment Patterns and a new section for CI/CD integration with buildevents, which facilitates distributed tracing in CI/CD pipelines. Additionally, a significant update to Refinery, a trace-aware sampling proxy, includes the new Stress Relief feature to manage load effectively. The query function now clarifies that a "month" is set to 28 days for consistent data comparison. Furthermore, a new OpenTelemetry SDK for Python offers auto-instrumentation and improved trace visualization. The updates emphasize ease of use, precision, and enhanced data management for users, with more detailed information available in release notes and online resources.

Honeycomb's latest fundraising post highlights a significant $50 million Series D funding round, led by Headline and supported by existing and new investors, to bolster its efforts in enhancing observability tools for software engineering teams. The company reports consecutive years of revenue growth and expanded its customer base to include prominent names like Robinhood and Equinix. Honeycomb emphasizes its commitment to creating a company culture that employees are proud of and highlights its recent accolades from Comparably in various categories, such as "Best Company for Diversity" and "Best Company Culture." The post reflects on the growing industry momentum towards observability, citing recognition from Gartner and the popularity of OpenTelemetry as evidence of its mainstream acceptance. Honeycomb plans to invest in automatic instrumentation, making observability more accessible, and improving solutions for modern architectures while maintaining its unique approach to observability, which integrates logs, metrics, and traces seamlessly. The company expresses gratitude towards its community and team, crediting them for its success and looking forward to furthering the impact of its observability tools.

Honeycomb, a prominent observability platform, announced it has secured $50 million in Series D funding to further its growth and global expansion, bringing its total capital raised to nearly $150 million. This funding round, led by Headline with contributions from Insight Partners and Scale Venture Partners, follows a year of significant revenue growth and the addition of notable customers like Robinhood and Equinix. Honeycomb distinguishes itself in the tech industry by providing engineering teams with tools to quickly analyze high-cardinality data, enabling them to troubleshoot modern distributed systems effectively, without relying on legacy APM tools. The funding will support Honeycomb’s expansion and further development of its integration strategy, while also bolstering its support for OpenTelemetry. The platform's unique approach to observability, which sidesteps the data correlation problem across logs, metrics, and traces, is seen as essential for engineering teams facing complex cloud environments and economic challenges, allowing them to deliver faster and more reliable user experiences.

Participating in Jen Dary's podcast "Be Plucky," the author discussed the benefits of career coaching and the practice of timeboxing role changes, reflecting on personal experiences with career transitions and specialization shifts. The conversation highlighted the value of career coaches in uncovering new career paths and providing guidance, as well as the importance of building a supportive network of coaches and mentors, termed as "Manager Voltron." The author emphasized the effectiveness of timeboxing as a strategy to manage career experiments, allowing for growth and re-evaluation without commitment to permanent change. The podcast also explored the significance of self-care in career development, acknowledging the impact of personal well-being on professional success and addressing the challenges of imposter syndrome during career transitions. Ultimately, the author advocates for a balance between exploration and self-care, encouraging a proactive yet mindful approach to career advancement.