Improving query performance in Grafana Mimir: Why we dropped mmap from the store-gateway

Grafana Mimir, an open-source, scalable time series database, removed the use of mmap in its store-gateway component to enhance performance and stability. The store-gateway, responsible for fetching metrics data efficiently, previously used mmap to load index-headers, a technique that led to significant performance issues, including increased latency and health check timeouts due to Golang's inability to handle mmap's page faults effectively. The transition away from mmap involved adopting traditional file I/O operations and implementing several optimizations, such as buffered I/O, pooling file handles, and reducing unnecessary string creation, which collectively improved memory and CPU usage. These changes have virtually eliminated health check timeouts in Grafana Cloud Metrics clusters and are now available in the open-source version of Mimir as of v2.7. The development team plans to continue optimizing Mimir's components, including enhancing data streaming from object storage to the querier, with additional improvements to be discussed in future updates.