Internet-Scale Knowledge Retrieval: A Novel Vector Search Dataset at 10B Scale

Qdrant, collaborating with Vultr and Hugging Face, has released Qdrant-FineWeb-10B, an open-source vector-search benchmark built from the FineWeb corpus that contains more than 10 billion dense and sparse vectors, 24.47 TB of vector data, and 28.66 TB of source text and metadata. Addressing limitations in existing benchmarks, which typically cover at most 100 million embeddings and often lack exact ground truth or support for hybrid retrieval, the dataset includes exact top-1,000 nearest-neighbor results for 100,000 queries, calculated through more than one quadrillion GPU-accelerated distance computations. The release also includes Supernova, an open-source framework that automates embedding generation, brute-force ground-truth computation, database loading, and high-concurrency evaluation across systems such as Qdrant, Milvus, and Elasticsearch. Additional datasets target hybrid text retrieval through dense, sparse, and multi-vector representations and multimodal image-text retrieval, while Supernova’s SkyPilot-based distributed infrastructure is designed to run reproducibly across major cloud platforms, Kubernetes, and HPC clusters.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.