Grafana Loki query acceleration: How we sped up queries without adding resources

Grafana Loki has introduced a new feature in its 3.0 version called query acceleration, designed to speed up log queries without requiring additional computational resources. By leveraging n-grams and Bloom filters, this experimental feature aims to reduce the amount of irrelevant data processed during queries, enhancing efficiency and maintaining cost-effectiveness. Bloom filters, which allow for probabilistic matching with some false positives but no false negatives, help identify and skip non-matching data, thereby increasing query speed significantly. The new system also includes a query sharding strategy that uses Bloom filters to create fewer, fairer query shards, addressing uneven data distribution in traditional sharding. Although Bloom filters are especially effective for precise searches, not all queries will benefit from them, and certain conditions must be met for their application. This development stays true to Loki's principles of being schemaless and easy to use, with no additional configuration required for users when it becomes available on Grafana Cloud.