January 2022 Summaries

The Vespa Newsletter from January 2022 highlights several updates and enhancements to their platform, focusing on improved performance and functionality. Key updates include faster node recovery and re-balancing by doubling data migration throughput, configurable re-indexing speeds to optimize resource usage during schema changes, and the release of pyvespa 0.14.0, which introduces a retry strategy for various data operations and new deployment options. The newsletter also details enhancements to the Query API, such as improved support for weakAnd queries with unstructured input and better synonym expansion rules. Additionally, it discusses the switch from Optimaize to OpenNLP for language detection in documents and queries and introduces new blog posts on topics like ML model serving at scale and billion-scale nearest neighbor searches.

The blog post delves into the trade-offs involved in billion-scale vector search using Vespa, focusing on the balance between serving performance and accuracy in approximate nearest neighbor search. It discusses the use of the HNSW algorithm in Vespa to optimize vector search by employing the Microsoft SPACEV-1B dataset, which comprises over a billion int8 precision vectors. The experiment involves transforming the dataset into a binarized form, utilizing hamming distance for search, and evaluating the impact on performance and resource utilization with various HNSW parameters. The post highlights how enabling HNSW indexing can significantly reduce write throughput and increase memory usage, yet offers substantial speed improvements in search latency compared to brute force methods. Additionally, it explores the use of multithreaded search to mitigate latency challenges in exact nearest neighbor searches, and posits that while approximate methods show substantial speed gains, the choice between exact and approximate search depends heavily on specific use case requirements. The discussion sets the stage for future exploration of hybrid approaches to further optimize memory usage and search efficiency.

Vespa, a data platform, has significantly enhanced its data redistribution process, doubling its throughput and reducing the time required to replace a failing content node by half. These improvements are part of Vespa version 7.528.3 and involve several technical optimizations, such as enhanced scheduling semantics, asynchronous operations, and optimized handling of delete bucket operations, which collectively minimize latency spikes and bottlenecks. Specifically, the upgraded system now allows for an average throughput of 44 MB/sec during data redistribution, reducing the process duration significantly from 3 hours and 50 minutes to about 2 hours. These advancements ensure that data redistribution occurs with minimal disruption to query or write traffic and are crucial in maintaining data redundancy and system reliability in Vespa Cloud.

Machine-learned model serving at scale often encounters challenges when managing concurrent requests, particularly due to default thread settings that lead to resource contention and increased latency. Platforms like Vespa.ai, which use ONNX Runtime for model acceleration, demonstrate these issues with models like BERT-base, showing significant performance degradation as concurrent request numbers increase. The solution involves adjusting the threading model to allow each model evaluation to run sequentially in its own thread, thereby reducing intra-operation thread competition. Additionally, model distillation techniques, such as using XtremeDistilTransformers, offer a way to maintain accuracy while reducing computational complexity and improving throughput. Despite these optimizations, understanding the nuances of latency promises is crucial, as high concurrency can drastically affect real-world performance beyond advertised capabilities.