Mimir’s next-gen architecture—Kafka in the middle, object storage underneath, and a whole lot less coupling

Mimir, an open-source, time series database used by Grafana Cloud, underwent a significant architectural shift to improve efficiency and reduce costs. The new design introduces Kafka between the read and write paths, allowing for decoupling and improved resilience by employing a predictable partitioning scheme. This change reduces the total cost of ownership by 25% compared to the previous architecture, which relied heavily on replication and quorum reads/writes inspired by Cassandra. The implementation of WarpStream helps mitigate cross-availability-zone costs by utilizing object storage for data replication, trading reduced network transfer costs for a latency penalty. This architectural overhaul not only lowered costs but also fostered a culture of experimentation within the team, allowing them to test new storage solutions in parallel with existing ones. Despite the improvements, scalability remains consistent with the previous architecture, which was already highly scalable.