The Evolution of Sampling in Honeycomb: Introducing Refinery 2.0
Blog post from Honeycomb
Honeycomb's Refinery has been updated to version 2.0 to address challenges customers faced with managing vast telemetry data and configuring the tool effectively. Refinery acts as a sampling proxy, selecting representative telemetry samples to forward to Honeycomb, thus reducing redundant and costly data transmissions. The update includes a complete overhaul of the configuration structure, making it more user-friendly by standardizing expressions, improving defaults, and adding a version marker for future updates. A new conversion tool, `convert`, has been introduced to seamlessly transition existing configuration files to the new format, accompanied by metadata that aids in generating documentation. The update also incorporates strict validation to ensure consistency and correctness of configuration files, supports multiple metrics targets including OpenTelemetry, and fixes a crucial bug in dynamic samplers to accurately count spans, requiring possible adjustments in configuration values. Overall, Refinery 2.0 is designed to enhance ease of configuration, operational stability, and effectiveness in telemetry management, encouraging users to migrate to the new version for an improved experience.