August 2023 Summaries

Mall food courts and Honeycomb share a fondness for sampling, a practice that Honeycomb not only recommends to its customers but also uses internally, particularly through its Refinery tail-based sampling proxy. Sampling, which is inherently lossy, requires careful consideration to ensure that an organization's critical measurements, such as Service Level Objectives (SLOs) and triggers, are not adversely impacted. To manage this, the Heinrich Hartmann Sampling Error Calculator can be used to estimate the margin of error introduced by sampling. The process involves analyzing data, such as the sample rate and request rate, and adjusting parameters to calculate relative error counts for both compliance and budget burndown. The findings help determine the reliability of alerts and guide decisions on alert trustworthiness. If the margin of error is too high, adjustments in sampling rates, SLI formulas, or trigger targets may be needed. The text underscores the importance of balancing trace integrity and cost efficiency while adapting to changes in business goals and instrumentation, emphasizing the need for ongoing evaluation and refinement of sampling practices.

Tracetest has released an integration with Honeycomb, enabling users to leverage their existing distributed traces for comprehensive end-to-end and integration testing. Tracetest is a tool designed for trace-based testing, allowing users to define tests and assertions for microservices, utilize multiple transaction triggers, and ensure both system responses and processes function as intended. By integrating with Honeycomb, which provides observability capabilities, users can conduct trace-based tests by leveraging their OpenTelemetry-instrumented systems. This integration facilitates the identification of issues through trace data exploration, which can inform the creation of new tests and assertions, promoting a cycle of continuous improvement. Users can monitor trace-based tests over time with Honeycomb, gaining insights into system performance and the impact of changes, while Tracetest's capabilities in asserting against trace data ensure system functionality and help catch regressions. The integration provides a feedback loop for enhancing testing processes, enabling a detailed analysis of application behavior, and offering granular insights into system performance over time.

The Accelerate State of DevOps Report identifies four key metrics, known as DORA metrics, that separate high-performing software organizations: deployment frequency, lead time for changes, time-to-restore, and change fail rate. These metrics create a reinforcing feedback loop where improvements in one area can lead to enhancements in others. Observability plays a crucial role in this cycle by enabling a deeper understanding of deployments, thereby reducing time-to-restore and change failure rates. By integrating observability into development processes, organizations can rapidly identify and resolve issues, leading to more frequent and reliable deployments. The report suggests that starting with improving lead time and implementing observability can help organizations overcome low confidence and high overhead in deployments. Observability not only aids in understanding changes but also boosts confidence, reduces deploy overhead, and enhances the overall effectiveness of the DORA metrics, ultimately leading to faster, more reliable software delivery.

Honeycomb is a tool designed for software developers to quickly address production issues by providing fast query responses, even if it sacrifices a slight degree of data completeness. In contrast, compliance and audit workloads demand 100% data completeness over extended periods, often at the expense of query performance. Honeycomb retains telemetry data for 60 days, which is generally sufficient for most organizations, but some require indefinite retention. This text explores an experimental method to achieve infinite retention using Amazon S3, Amazon Athena, and AWS Glue, which allows storing raw telemetry data and querying it over long periods. While this approach offers a solution for long-term data retention and compliance, it results in slower data querying compared to Honeycomb, which is optimized for urgent and iterative use cases. The method involves using OpenTelemetry to export data to an S3 bucket, setting up Glue to structure the data, and using Athena to query it, albeit with a more complex and less user-friendly experience.

The W3C trace context specification represents a significant advancement in standardizing telemetry correlation for microservices, but it lacks guidelines on trace propagation limits, leading to potential issues such as unwanted telemetry effects and handling of Personally Identifiable Information (PII). In the .NET environment, trace propagation is managed through DistributedContextPropagator and OpenTelemetry's TextMapPropagator, which can be overridden to control trace data flow. The text illustrates how to disable inbound trace propagation by overriding these classes, emphasizing the importance of careful management of trace headers to avoid processing unwanted or sensitive data. The document highlights the complexity and responsibility involved in debugging distributed systems, suggesting a need for strategic decisions on handling trace data and hinting at more advanced approaches to be discussed in subsequent posts.

Tech debt, a contentious topic among engineering teams, often leads to misunderstandings due to its broad and vague nature, which can obscure the value of engineering work to business stakeholders. Engineers frequently feel they lack time to address tech debt, while product partners and business leaders question its importance, creating a conflict that often stems from a lack of clear communication. To mitigate this, the author suggests breaking down tech debt into more specific categories, allowing engineers to more effectively communicate the business value of their work. Tool chain investments, reliability, and rework are examples of non-feature work that can be challenging to prioritize but are essential for long-term success. Using frameworks like MEDDIC for tool adoption and setting Service Level Objectives (SLOs) for reliability can help align technical teams with business goals. While the term "tech debt" may not always be helpful in advocating for work, the concept of debt itself remains valuable in making decisions about when to invest in engineering efforts. By using more precise labels that highlight the business benefits, stakeholders can better understand the necessity of such investments, leading to more productive discussions and alignment on priorities.

The OpenTelemetry Go project has reached a significant milestone with the introduction of automatic instrumentation via eBPF, which simplifies data generation from Go applications without the need for redeployment. The agent, currently in beta, supports net/http, gRPC, gin, and MUX instrumentation and can be used on bare metal or in Kubernetes environments. eBPF allows the agent to run sandboxed programs, or probes, capable of inspecting and instrumenting calls to supported libraries in Go applications, which compile directly into machine code and cannot be patched like bytecode languages. This advancement aligns with eBPF's goal of enabling runtime inspection in a language-agnostic manner, and the OpenTelemetry Go project, supported by contributors like Honeycomb, aims to extend library support and explore further eBPF applications to facilitate automatic instrumentation across different programming environments.