The Quest for One Million IOPS: Benchmarking Storage at LanceDB

The blog post discusses the challenges and methodologies involved in optimizing the Lance file format for achieving high IOPS (Input/Output Operations Per Second), particularly aiming for 1,000,000 IOPS using NVMe drives. The author explores the intricacies of random access, commonly used in search workflows, and explains the importance of benchmark tests that consider both latency and throughput. The post highlights the complexities of achieving high IOPS due to synchronization overhead and system calls and details the process of reworking the scheduler and implementing io_uring to improve performance. Initial benchmarks on various hardware setups revealed bottlenecks, leading to iterative improvements in task management and I/O operations. Through a combination of a new scheduler and io_uring, the system achieved 1,500,000 IOPS, though further work is required to maintain high recall and address compute-storage separation in future architectures. The author plans to explore additional workflows, such as key-value stores, to optimize query performance further.