March 2023 Summaries

Organizations are increasingly utilizing deep-learned embeddings for search and recommendation applications, facing challenges in managing these embeddings efficiently in production. An emerging strategy to address these challenges is the use of frozen foundational embeddings, which are reusable and can be tailored for specific tasks, thereby reducing the complexity and cost typically associated with embedding lifecycle management. Vespa serves as a platform that supports the customization of these frozen embeddings, allowing developers to modify the query tower in two-tower embedding models while keeping the document tower frozen. This approach significantly reduces the computational and storage costs involved in maintaining multiple embedding models. By using frozen embeddings, developers can streamline the process of deploying and evaluating new models without the need to reprocess data, facilitating a more efficient and scalable system for handling large-scale datasets. The integration of Vespa enables the execution of advanced transformations using deep neural networks, enhancing the flexibility and personalization of applications. This approach not only simplifies the infrastructure required for managing embeddings but also allows for frequent model deployment and evaluations, thereby increasing the efficiency and effectiveness of machine learning operations in production environments.

Vespa has introduced multi-vector HNSW indexing, a feature that enhances semantic search by allowing multiple vectors per document, thus addressing the limitations of single-vector representation, particularly for lengthy text documents like Wikipedia articles. This advancement in Vespa, available from version 8.144.19, leverages deep learning models to represent data in high-dimensional vector spaces, enabling more efficient and precise nearest neighbor searches. Multi-vector indexing mitigates challenges such as input length limitations of Transformer-based models, which typically necessitate chunking text into smaller segments for effective embedding. By facilitating such indexing, Vespa simplifies the management of complex data structures, supports diverse retrieval tasks across various domains, and enhances search accuracy without requiring intricate relationship modeling. This development is particularly beneficial for applications requiring multi-modal searches, such as e-commerce, where products have evolving metadata and multiple associated vectors like images. The implementation demonstrates minimal performance differences between single and multi-vector indexing, with only slight increases in feed time and query latency, thus offering a scalable and efficient solution for deploying advanced semantic search capabilities.

Vespa Cloud has introduced support for private regional endpoints in both AWS and GCP, allowing users to connect to their Vespa applications through the private network of their chosen cloud provider instead of using public internet endpoints. This enhancement reduces latency and costs while providing an additional layer of security by enabling users to disable public endpoints. To utilize this feature, clients must be in the same region as the Vespa clusters they interact with, and the setup involves configuring a private endpoint service, setting up a VPC endpoint within the same region, and verifying the connectivity. Further details can be found through AWS PrivateLink or GCP Private Service Connect.

Vespa's March 2023 newsletter highlights recent advancements and features in their platform, particularly focusing on GPU-accelerated machine learning inference which enhances performance and cost efficiency by supporting various models like TensorFlow and ONNX. The newsletter also discusses Vespa Cloud's business continuity plan (BCP)-aware autoscaling, allowing applications to define their BCP traffic structure explicitly and manage resource allocation efficiently during zone failures. Vespa continues to enhance its e-commerce applications by integrating advanced NLP techniques for query suggestions, hybrid ranking, and vector search, which improves user experience through semantic search and contextual navigation. Furthermore, Vespa has improved its static memory usage and made document extraction more accessible with the Vespa CLI, alongside splitting its Python experimentation library, pyvespa, into two repositories to better support data scientists. The platform now supports GCP Private Service Connect, facilitating private and secure application access, and allows deployment across AWS and GCP regions to enhance cloud provider redundancy and streamline migration projects.

Vespa has introduced GPU-accelerated ONNX model inference in Vespa Cloud, offering enhanced performance and cost efficiency compared to CPU instances. Users can configure GPU instances in AWS zones through the services.xml file, with automatic provisioning and configuration by Vespa Cloud. GPU support is also available for open-source Vespa, requiring specific container configurations. A benchmark using the CORD-19 application demonstrated that GPU instances significantly reduce latency, achieving an average latency of 212 ms and throughput of 18.8 QPS, compared to CPU instances with 1011 ms latency and 3.95 QPS, while also being 13% more cost-effective. This highlights the potential advantages of GPU acceleration for machine learning model inference in Vespa.