How We’re (Ab)using Hashicorp’s Consul at Grafana Labs

Grafana Labs, after years of leading Cortex development, has transitioned to using Grafana Mimir for scalable metric storage, but continues to utilize Hashicorp's Consul for service discovery and data distribution. Although Consul's KV store was initially useful, scaling operations revealed limitations, prompting Grafana Labs to implement unconventional solutions like compacting every 128 versions and using memory disks to optimize performance. These adjustments allowed for greater scalability and consistency, albeit at the cost of reliability, by operating with a single-node Consul setup. To manage API load, rate limits were introduced, and the team plans to incorporate the gossip protocol to further reduce dependency on Consul. Despite these "abuses," Grafana effectively adapted Consul to meet its operational needs.