Billion-scale vector search using hybrid HNSW-IF

The blog post discusses a cost-efficient hybrid approach for billion-scale vector search using a method called HNSW-IF, which combines Hierarchical Navigable Small World (HNSW) with a disk-backed inverted file system. This method addresses the high memory requirements of in-memory algorithms like HNSW by utilizing solid-state disks to store most vector data while maintaining in-memory graph data structures for efficient search. Inspired by the SPANN approach, the Vespa team implemented this hybrid method, which partitions a vector dataset into clusters, using centroids to represent clusters, and stores non-centroid vectors in disk-based data structures. The Vespa implementation features real-time indexing and querying capabilities and supports CRUD operations, making it suitable for real-world applications with varying query volumes. Experiments show that the hybrid HNSW-IF system can achieve high recall rates with acceptable query latencies, making it a viable solution for applications requiring scalable, accurate vector search without the high costs associated with fully in-memory systems.