May 2019 Summaries

Rachel's guide on dynamic sampling delves into techniques for varying sample rates to maintain a target collection rate while preserving key events for debugging. The document explores the implementation of dynamic sampling in Go, a process that can be adapted to any language supporting hashes, pseudorandom number generation, and concurrency. It critiques naive fixed-rate sampling, suggesting alternatives like consistent sampling and target rate sampling, which automate sample rate adjustments based on incoming request rates. The guide further discusses sampling by event properties to capture outliers such as errors or high latency events, and emphasizes the importance of consistent sampling decisions across traces. It introduces a buffered sampling approach for tail sampling, which allows for retrospective decisions on trace retention. Ultimately, the guide presents a comprehensive strategy for dynamic sampling, integrating head and tail per-key target rate sampling to optimize the debugging process in high-throughput systems, while promoting Honeycomb's buffered sampling feature for enhanced trace analysis.

Honeycomb utilizes Observability-Driven Development to address and prevent database load issues, a common challenge for growing online services. As the service's popularity increases, so does the database load, prompting the need for strategies like caching, read replicas, and sharding. Honeycomb has already navigated this cycle twice, each time aiming for significant growth before addressing further improvements. The company monitors database utilization, setting a 40% threshold for taking corrective action. Past solutions have included adding caches and upgrading RDS instance types, and they are now exploring the potential of extending cache TTLs to reduce load. By analyzing database query data with Honeycomb’s tools, they identified key areas to optimize, achieving significant growth with minimal effort. This approach allows Honeycomb to consider future strategies for scaling without being constrained by database limitations.

Engineer John Obelenus clarifies the distinction between technical debt and escalating risk in his article "Not Everything is Tech Debt," emphasizing the importance of understanding and managing these concepts in complex systems. He highlights that unexpected errors are inevitable in complex systems and suggests using tools like Honeycomb to address these issues effectively. Obelenus advises that lacking necessary data to resolve unexpected errors contributes to increased risk, and recommends Honeycomb as a valuable resource for those encountering such challenges.