Getting At The Good Stuff: How To Sample Traces in Honeycomb

Sampling is an essential technique for managing the infrastructure and telemetry systems of large-scale applications by reducing the amount of data retained to a statistical sample rather than capturing all requests, thereby controlling costs and resource utilization. This article is intended for developers and operators already using Honeycomb instrumentation, offering insights into various sampling methods, including random, deterministic, rule-based, and dynamic sampling, along with their respective advantages, drawbacks, and appropriate use-cases. While sampling helps lower the costs associated with instrumentation, it cannot eliminate them entirely, as the process involves additional CPU usage, memory allocation, and network bandwidth. Honeycomb supports different sampling techniques, such as head-based and tail-based sampling, to manage traces effectively, though each comes with its own set of considerations regarding request latency and trace completeness. The article also provides troubleshooting advice for common issues like increased request latency or missing trace spans and offers resources for further reading and exploration of sampling strategies.