Scaling Ingest With Ingest Telemetry

The introduction of Environments & Services in Honeycomb has led to an increase in the creation of smaller datasets, posing scaling challenges that were addressed by leveraging Honeycomb's telemetry to optimize data distribution across Kafka partitions. As events in Honeycomb are organized into datasets corresponding to services, load balancing among the Kafka partitions is crucial, and when a dataset's traffic grows, additional partitions are added to manage the load. SRE Fred Hebert developed the "Fractional Partition Weight" measure to identify the best datasets to expand, ensuring efficient traffic distribution while avoiding overloading other partitions. The approach considers the diminishing returns of adding more partitions, emphasizing the need for linear increases to effectively manage traffic. Originally, manual analysis and admin UI adjustments were used to manage dataset partition changes, but with the new Query Data API, Honeycomb improved its tool to automate recommendations based on comprehensive data from their dogfood environment, enhancing reliability and reducing toil. The solution illustrates the complexities of scaling partitioned setups and highlights Honeycomb's commitment to improving observability tools, inviting users to explore their free tier and recent feature enhancements.