April 2024 Summaries

Vespa has introduced support for Matryoshka Representation Learning (MRL) and Binary Quantization Learning (BQL) in its native hugging-face embedder, allowing for significant reductions in vector search costs by encoding text as binary vectors instead of large float vectors. These techniques, which can be applied as post-processing steps after model inference, facilitate the creation of compact text embeddings that reduce storage and computational resources while maintaining about 90% of the accuracy of the original float-based embeddings. The adoption of these methods within Vespa enables cost-effective and scalable vector search solutions, particularly advantageous for unstructured data and scenarios requiring large-scale data processing. This move not only slashes storage costs but also enhances the speed of similarity searches by utilizing efficient distance metrics like Hamming distance for binary vectors, thereby supporting more complex retrieval and ranking tasks without compromising performance.

In the April 2024 Vespa Newsletter, significant updates and advancements are highlighted, including the introduction of the SPLADE Embedder, which enhances learned sparse retrieval by using term impact scores from large language models, and the support for half-precision floating-point (float16) ONNX models, improving inference performance on GPUs. New guides are available for using Cohere embedding models, which now support binary and int8 vectors, offering cost savings and performance benefits. The newsletter also discusses enhancements to the ColBERT embedder, allowing multi-paragraph inputs and highlights recent blog posts and case studies showcasing Vespa's application in semantic search and recommendation systems across various companies. Additionally, it announces upcoming meetups and conferences focused on improving the usefulness of large language models and retrieval augmented generation.

Marqo, a vector database, has selected Vespa as its preferred platform after extensive benchmarking against other open-source and proprietary databases like Milvus, OpenSearch, Weaviate, Redis, and Qdrant. Vespa was chosen due to its superior performance, with internal tests showing it significantly reduced latency and increased throughput compared to Marqo's previous version backed by OpenSearch. The discussion highlights common oversights in benchmarking vector databases, particularly the need to evaluate performance on large datasets and under conditions of concurrent indexing and search, as well as space complexity and high availability. Vespa's capabilities make it a strong contender for high-throughput production vector search applications, outperforming alternatives like Elasticsearch in several key areas.

Satoshi Takatori describes Stanby's migration from multiple search engines to a unified system using Vespa, aiming to address various challenges with their existing setup. Stanby, a major Japanese job search engine, faced issues with its current systems: Solr-based ABYSS for organic search and Elasticsearch for advertisement search, including risks of service unavailability, development constraints, and scaling difficulties. By choosing Vespa, an open-source big data serving engine known for its real-time, low-latency, and high-throughput capabilities, Stanby aims to consolidate resources, improve search accuracy, and enhance operational independence. The migration process involved thorough investigation, functional verification using Vespa's Docker image, and a multi-node Vespa cluster setup on AWS for high availability. While the organic search migration is complete, the advertisement search is underway, with future plans to leverage Vespa's capabilities in vector search and auto-scaling, despite challenges such as language processing and the need for custom solutions in Japanese.