September 2021 Summaries

Open Neural Network Exchange (ONNX) is an open format that represents machine learning models, enabling AI developers to use models with various frameworks, tools, runtimes, and compilers. Milvus is an open-source vector database designed for massive unstructured data analysis. This article introduces how to use multiple models for image search based on ONNX and Milvus, using VGG16 and ResNet50 models as examples. The process involves converting pre-trained AI models into the ONNX format, extracting feature vectors from images using these models, storing vector data in Milvus, and searching for similar images based on Euclidean distance calculations between vectors.

"DiskANN: A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale Dataset" is a paper published in NeurIPS 2019 that introduces an efficient method for index building and search on billion-scale datasets using a single machine. The proposed scheme, called DiskANN, builds a graph-based index on the dataset SIFT-1B with a single machine having 64GB of RAM and a 16-core CPU, achieving over 95% recall@1 at more than 5000 queries per second (QPS) with an average latency lower than 3ms. The paper also introduces Vamana, a new graph-based algorithm that minimizes the number of disk accesses and enhances search performance. DiskANN effectively supports search on large-scale datasets by overcoming memory restrictions in a single machine.

Mengjia Gu, a data engineer at Zilliz and open-source community member of Milvus, discusses the application of vector databases in DNA sequence classification. Traditional sequence alignment methods are unsuitable for large datasets, making vectorization a more efficient choice. The open-source vector database Milvus is designed to store vectors of nucleic acid sequences and perform high-efficiency retrieval, reducing research costs. By converting long DNA sequences into k-mer lists, data can be vectorized and used in machine learning models for gene classification. Milvus' approximate nearest neighbor search algorithm enables efficient management of unstructured data and recalling similar results among trillions of vectors within milliseconds. The author provides a demo showcasing the use of Milvus in building a DNA sequence classification system, highlighting its potential applications in genetic research and practice.