December 2023 Summaries

Andrey Vasnetsov, Co-founder and CTO at Qdrant, discusses the intricacies of vector search and its applications in a session with Learn NLP Academy. He explores the functionality of the Qdrant search engine and the Quaterion similarity learning framework, emphasizing the role of similarity learning in multimodal settings. Vasnetsov contrasts Elastic search embeddings with vector search engines, highlighting the support for multiple embeddings. He also touches on the topics of fundraising, venture capital discussions, and his vision for the evolution of vector search, including the significance of fine-tuning for out-of-domain scenarios.

Qdrant 1.7.0 introduces several new features and improvements to enhance its capabilities as a vector search engine. This release includes support for sparse vectors, allowing for more efficient keyword-based searches, and a new Discovery API designed to refine search parameters for precision and versatility. The update also adds user-defined sharding, enabling users to determine how data is distributed across nodes, and a snapshot-based shard transfer method to improve data mobility within clusters. These features aim to provide greater control and flexibility in managing and searching data, enhancing overall system performance. Additionally, minor improvements address performance issues, reduce latency, and stabilize operations in high-load environments. Qdrant continues to evolve based on user feedback, encouraging community engagement to drive future development.

Sparse vectors offer a streamlined approach to text representation by focusing on key features and minimizing non-essential elements, which is particularly beneficial in information retrieval systems like search and recommendation engines. In contrast to dense vectors that capture nuanced relationships with non-zero values, sparse vectors excel in efficiency and interpretability, especially in domains with rare or specialized terms. The SPLADE model is highlighted as a flexible method for generating sparse vectors, using a transformer architecture to enhance retrieval capabilities by expanding terms contextually relevant to queries. This method offers advantages over traditional models like BM25 by addressing its limitations and reducing memory usage, making it suitable for large-scale systems. Qdrant supports hybrid search by allowing the integration of both dense and sparse vectors within the same collection, facilitating efficient and accurate retrieval processes. This hybrid approach combines the strengths of both vector types, ensuring comprehensive search results and offering various strategies for result fusion to enhance search accuracy.