Scaling Prometheus: How we’re pushing Cortex blocks storage to its limit and beyond

Grafana Labs, after ceasing contributions to Cortex, has launched Grafana Mimir as an open-source long-term storage for Prometheus, enabling scalability to over 1 billion metrics. This blog post explores the challenges and solutions encountered in scaling Cortex's blocks storage, such as inefficiencies in querying non-compacted blocks and the need for vertical and horizontal compaction to optimize query performance. The Cortex architecture uses a hash ring for sharding and replication, allowing for dynamic scaling and efficient distribution of workloads across store-gateways. The compactor's single-threaded nature poses scalability limitations, which are mitigated by per-tenant sharding and other optimizations. Future improvements will focus on further query optimizations and shuffle sharding to enhance the system's scalability. The work represents a collaborative effort within the Cortex and Prometheus communities, highlighting Grafana Labs' ongoing commitment to improving metrics storage solutions for large-scale deployments.