How WarpStream Powers Grafana Labs’ Redesigned Architecture

Grafana Mimir, an open-source, horizontally scalable time series database (TSDB), is marking its third anniversary as a leading Prometheus-compatible metrics backend with over 4.6k GitHub stars. As part of its evolution, Grafana is making architectural changes to Mimir and Grafana Cloud Metrics to enhance reliability and support future growth. The current architecture, which involves stateful ingester nodes with in-memory TSDB and a write-ahead log (WAL), poses challenges such as disrupted writes by heavy queries and complex management. To address these, Grafana is redesigning Mimir by decoupling the read and write paths using Apache Kafka, and for Grafana Cloud Metrics, exploring WarpStream. WarpStream is a Kafka-compatible data streaming platform built on object storage, offering cost-effective multi-availability zone deployments without inter-AZ networking costs and providing stateless, auto-scaling benefits. Load tests confirmed WarpStream's capability to handle Grafana Cloud Metrics' scalability needs, and the new architecture is gradually being rolled out, promising enhanced resilience to traffic spikes and ingest volume. Once the rollout is stable, the new architecture will be fully implemented in Mimir's open-source version.