Introducing Gridstore: Qdrant's Custom Key-Value Store

Qdrant developed its own custom key-value storage system named Gridstore to address specific issues encountered with RocksDB, such as latency spikes due to its compaction process and the complexity of tuning options. Gridstore is designed to efficiently handle sequential keys and variable-sized data, featuring a three-layer architecture consisting of a Data Layer for fast retrieval, a Mask Layer for space reuse without expensive compaction, and a Gaps Layer for effective updates. This architecture allows for rapid lookups and efficient space management, reducing computational overhead and ensuring data integrity through a Write-Ahead Log (WAL) system. Rigorous testing, including model testing and crash resilience checks, demonstrated Gridstore's reliability and performance, with benchmarks showing significant improvements in data ingestion speed and consistent throughput compared to the previous storage solution. While currently integrated with Qdrant for payloads and sparse vectors, there are plans to potentially release Gridstore as a standalone contribution to the Rust community once the API stabilizes.