Company
Date Published
Author
Joe Elliott
Word count
977
Language
English
Hacker News points
None

Summary

At Grafana Labs, the integration of Scylla, an open-source Cassandra-compatible backend, with Jaeger tracing has demonstrated impressive performance capabilities, handling approximately 175,000 writes per second despite being deployed with default settings on Google's Kubernetes Engine. The disparity between the high number of writes and the lower span ingestion rate, around 8,000 spans per second, is attributed to the indexing process in Jaeger, which writes multiple rows per span to support detailed search capabilities. To manage backend load while maximizing span ingestion, Grafana Labs recommends adjusting sampling rates using Jaeger's remote sampler and utilizing configuration options to control indexing, thereby reducing unnecessary writes. Implementing a tag blacklist, disabling certain indexes, and using "firehose" mode for spans with less indexing needs have been effective strategies, resulting in a significant reduction in backend writes. Overall, leveraging these configurations has allowed Grafana Labs to maintain high performance and stability in their distributed tracing infrastructure.