Dynamic Sampling by Example

Rachel's guide on dynamic sampling delves into techniques for varying sample rates to maintain a target collection rate while preserving key events for debugging. The document explores the implementation of dynamic sampling in Go, a process that can be adapted to any language supporting hashes, pseudorandom number generation, and concurrency. It critiques naive fixed-rate sampling, suggesting alternatives like consistent sampling and target rate sampling, which automate sample rate adjustments based on incoming request rates. The guide further discusses sampling by event properties to capture outliers such as errors or high latency events, and emphasizes the importance of consistent sampling decisions across traces. It introduces a buffered sampling approach for tail sampling, which allows for retrospective decisions on trace retention. Ultimately, the guide presents a comprehensive strategy for dynamic sampling, integrating head and tail per-key target rate sampling to optimize the debugging process in high-throughput systems, while promoting Honeycomb's buffered sampling feature for enhanced trace analysis.