Indexing All of Wikipedia on a Laptop

Cohere has released a dataset containing all of Wikipedia chunked and embedded to vectors using their multilingual-v3 model. This makes creating a semantic, vector-based index of Wikipedia practical for an individual for the first time. The JVector library now supports indexing larger-than-memory datasets by performing construction-related searches with compressed vectors. By using Locally-Adaptive Quantization (LVQ) compression, it improves on previous methods and allows for faster searches while maintaining accuracy. This has enabled the indexing of all of English Wikipedia on a laptop, which was previously not practical.