January 2020 Summaries

In Ben Linder's InfoQ article, Honeycomb engineer Ben Hartshorne is featured for his work on enhancing the software supply chain by reducing build times through observability. The article delves into how Hartshorne gained a deeper understanding of the build process performance, which provided significant benefits and valuable insights.

Feature flags are useful for managing code changes, but long-lived ones can lead to technical debt by complicating code and creating unused sections. Honeycomb's trigger alerts can help manage this by notifying developers when it's time to clean up these flags. An example is provided where new code in a Secure Tenancy service was incompatible with existing web app code, necessitating a feature flag. To address the issue of remembering to remove outdated flags, Honeycomb's integration is used to derive version numbers from user_agent strings, enabling developers to set up alerts when certain conditions are met, such as all users upgrading past a particular version. This approach allows for the proactive management of feature flags, reducing the risk of forgotten tech debt and demonstrating that alerts can be used for preventive maintenance, not just emergencies.

Sampling is an essential technique for managing the infrastructure and telemetry systems of large-scale applications by reducing the amount of data retained to a statistical sample rather than capturing all requests, thereby controlling costs and resource utilization. This article is intended for developers and operators already using Honeycomb instrumentation, offering insights into various sampling methods, including random, deterministic, rule-based, and dynamic sampling, along with their respective advantages, drawbacks, and appropriate use-cases. While sampling helps lower the costs associated with instrumentation, it cannot eliminate them entirely, as the process involves additional CPU usage, memory allocation, and network bandwidth. Honeycomb supports different sampling techniques, such as head-based and tail-based sampling, to manage traces effectively, though each comes with its own set of considerations regarding request latency and trace completeness. The article also provides troubleshooting advice for common issues like increased request latency or missing trace spans and offers resources for further reading and exploration of sampling strategies.

Honeycomb has successfully leveraged AWS Lambda to enhance its storage engine, utilizing the Serverless framework and Python to build a TODO API integrated with DynamoDB for demonstration purposes. To achieve robust instrumentation and tracing in this complex system, the company installed Honeycomb's Beeline library, enabling efficient data processing and tracing within the Lambda environment despite its unique constraints, such as cold starts and container lifecycle challenges. The article provides a detailed walkthrough on setting up the application, implementing tracing with Beeline, and deploying the serverless app with Cloudwatch logs for reliable telemetry without significant performance penalties. Additionally, it explores distributed tracing, demonstrating how to link Lambda instrumentation with application-wide traces using context passing and the X-Honeycomb-Trace header, ultimately enhancing visibility and performance monitoring across serverless applications.

Honeycomb has introduced Service Level Objectives (SLOs) to help users better manage and understand their systems' performance by quantifying the success rate of Service Level Indicators (SLIs). This approach allows users to categorize events as passing, failing, or irrelevant and to set targets for passing rates over specific periods, such as 30 days. By implementing error budgets, Honeycomb provides a way to monitor and manage transient failures and minor running errors, offering insights into how these affect overall system performance. Additionally, Honeycomb's SLO functionality includes Burn Alerts, which notify users when their error budget is at risk of being exhausted, and provides tools like Heatmaps and BubbleUp charts to diagnose and address issues. These SLOs aim to assist organizations in maintaining service reliability and making informed decisions about their operational strategies. Honeycomb's SLOs are available to Enterprise customers and can be explored via a free trial for new users.