Indexing sparse vectors with Turso

The recent 0.3.0 release of the Turso database introduces significant enhancements for vector search, particularly for sparse vectors, by improving their storage representation and indexing methods. With the implementation of SIMD acceleration for dense vectors and efficient indexing for sparse vectors, Turso now supports faster search capabilities, including a 21% speedup for approximate searches. Additionally, the update focuses on indexing sparse vectors using the Weighted Jaccard distance, which offers substantial space savings and improved search efficiency. The database leverages SQLite's VTable concept and introduces an Index Method feature that allows custom data access methods to be integrated into the query planner, facilitating better developer experience (DX). By building an inverted index and using frequency-based component selection, Turso optimizes search performance, especially for sparse vectors, which are prevalent in fields like bioinformatics and text mining. Benchmarks show that Turso's indexing strategies significantly outperform brute-force methods, especially under different selectivity profiles, by effectively balancing speed and accuracy through adaptive length filtering and component selection optimization.