Universal I/O in Qdrant

Qdrant developed Universal I/O to allow its vector database to use the same data-structure and search code across local disk storage and remote object storage used in serverless deployments. Previously, Qdrant relied on memory-mapped files, which are fast for local cached data but unsuitable for network-backed reads that can take tens of milliseconds and fail in ways disk reads do not. Rather than making all code asynchronous or maintaining separate implementations, the team created a synchronous ReadPipeline abstraction that schedules reads and collects completions while allowing each backend to use an appropriate model: immediate mmap reads, concurrent io_uring requests, or asynchronous network operations bridged through a dedicated Tokio runtime. The interface uses clone-on-write buffers so local mmap reads remain zero-copy while other backends can return owned data transparently. Qdrant also built DiskCache, which mirrors immutable or append-only remote files into local mmap-backed files in 16 KiB blocks, tracks cached blocks with a compressed bitmap, merges overlapping in-flight fetches, and returns local slices after remote data is retrieved. This architecture preserves efficient local behavior while enabling concurrent serverless object-storage access and allowing future storage backends to be added without rewriting Qdrant’s core data structures.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.