November 2021 Summaries

This article introduces an open-source vector database, Milvus, paired with PaddlePaddle, a deep learning platform, to address the issues faced in developing recommender systems. The basic workflow of a recommender system involves candidate generation and ranking stages. The product recommender system project uses three components: MIND (Multi-Interest Network with Dynamic Routing for Recommendation at Tmall), PaddleRec, and Milvus. MIND is an algorithm developed by Alibaba Group that processes multiple interests of one user during the candidate generation stage. PaddleRec is a large-scale search model library for recommendation, while Milvus is a vector database featuring a cloud-native architecture used for vector similarity search and vector management in this project. The system implementation involves data processing, model training, model testing, generating product item candidates, and data storage and search.

In the era of Big Data, unstructured data represents roughly 80-90% of all stored data. Traditional analytical methods fail to pull out useful information from these growing data lakes. To address this issue, researchers are focusing on building general-purpose vector database systems that can handle high-dimensional vector data and support advanced query semantics. The article discusses the design and challenges faced when building such a system, including optimizing cost-to-performance ratio relative to load, automated system configuration and tuning, and supporting advanced query semantics. It also introduces Milvus, an AI-oriented general-purpose vector database system developed by Zilliz's Research and Developement team.