October 2023 Summaries

This article discusses the use of vector search technologies, such as Milvus and Zilliz Cloud, in building retrieval augmented generation (RAG) applications. RAGs are question-answering applications that allow large language models (LLMs) to access a verified knowledge base for context. The article highlights various configuration choices that can affect the quality of retrieval, including data selection, embedding model, index type, amount of context retrieved, and chunk size. It also introduces TruLens, an open-source library for evaluating and tracking the performance of LLM applications like RAGs. By using TruLens to evaluate different configurations and parameters, developers can identify failure modes and find the most performant combination for their specific use case.

This tutorial demonstrates how to build a retrieval augmented generation (RAG) type app using LangChain and Milvus. The process involves reviewing LangChain self-querying, working with Notion docs in LangChain, ingesting Notion documents, storing them in a vector database, and querying the documents. The tutorial uses LangChain for operational framework and Milvus as the similarity engine. It covers how to load and parse a Notion document into sections to query in a basic RAG architecture, with future tutorials exploring different chunking strategies, embeddings, splitting strategies, and evaluation methods.

Large Language Models (LLMs) are driving innovation in AI, with LLM-driven Agents at the forefront. These agents combine LLMs with planning, memory, and tool modules to make decisions and take actions autonomously. The AutoGPT project demonstrates their potential by generating tasks, prioritizing them, and executing them using external resources. However, challenges such as getting stuck in loops and prompt length constraints need to be addressed. Ongoing research is focused on improving LLMs' reasoning abilities, enhancing agent frameworks, and developing specialized agent applications for various scenarios.

This tutorial explores the impact of different chunking strategies on retrieval augmented generation applications using LangChain. Chunking is the process of dividing text into smaller parts, and the choice of strategy can significantly affect the output quality. The code for this post can be found in a GitHub repo on LLM experimentation. The tutorial covers setting up the environment, importing necessary tools, and creating a function that takes parameters for document ingestion and chunking experimentation. It then tests five different chunking strategies with varying lengths and overlaps. The results show that finding an ideal chunking size is challenging and depends on the desired output format. Future tutorials may cover testing overlaps and using other libraries to refine chunking strategies further.

Over the past decade, the author has specialized in various aspects of data infrastructure, including access control, data privacy, NoSQL databases, and web-scale data indexing. In recent years, big data emerged as a significant innovation with technologies like MapReduce, distributed computing, and structured data storage leading the way. However, the AI era requires different technology stacks, especially with the growing popularity of Large Language Models. Embedding and vector stores are at the center of this stage, which is also the focus of Zilliz. The author's experience includes working on search indexing at Google, where they built ultra-flexible infrastructures to understand billions of images and videos on the public web. They believe that AI-native infrastructure holds the key to the future of business and are enthusiastic about democratizing this highly complex infrastructure for resource-limited startups. The author joined Zilliz due to its ambitious mission, exceptional team, and challenging work environment. At Zilliz, they build a suite of tooling and services that ease the information retrieval process on unstructured data, including Towhee, Akcio, and a vector database for efficient storage and search of vector embeddings.

Milvus introduces the MMap feature, which redefines how large data volumes are managed and promises cost efficiency without compromising functionality. MMap is a memory-mapped file technology that allows Milvus to map large files directly into system memory space, transforming them into contiguous memory blocks. This integration eliminates the need for explicit read or write operations, fundamentally changing how Milvus manages data. The feature benefits vector databases by enabling more efficient storage and access to large files or situations where users need to access files randomly. However, it may cause performance fluctuations as data volume grows. Enabling MMap in Milvus is straightforward, requiring a modification of the milvus.yaml file. Future updates will refine memory usage and provide more granular control over the feature.

This blog compares two vector databases, Qdrant and Zilliz/Milvus. While both are purpose-built for vector data, they serve different market needs. Qdrant is designed for developers who prioritize modern technology and minimal infrastructure maintenance, while Zilliz/Milvus is engineered for extreme scale, high performance, and low latency. The benchmark results show that Zilliz Cloud outperforms Qdrant Cloud in terms of queries per second (QPS), queries per dollar (QP$), and latency. Furthermore, the feature comparison highlights differences in scalability, functionality, and purpose-built features between the two vector databases.

This tutorial demonstrates how to use LlamaIndex, an open-source data retrieval framework, to improve the performance of a chatbot built with Zilliz Cloud. The primary challenge addressed in this project is integrating an existing Milvus collection into LlamaIndex while handling differences in embedding vector dimensions and metadata field usage. By using LlamaIndex as a query engine, the chatbot's retrieval capabilities are significantly enhanced, providing more accurate and relevant responses to user queries.

Milvus, an open-source vector database, has introduced NATS messaging integration in its latest version 2.3. This feature enhances the handling of substantial data volumes and complex scenarios compared to its predecessor, RocksMQ. NATS is a distributed system connectivity technology implemented in Go that supports various communication modes like Request-Reply and Publish-Subscribe across systems. Milvus 2.3 offers a new control option, mq.type, which allows users to specify the type of MQ they want to use. To enable NATS, set mq.type=natsmq. The migration from RocksMQ to NATS is seamless and involves steps like stopping write operations, flushing data, modifying configurations, and verifying the migration through Milvus logs. Performance testing results show that NATS outperforms RocksMQ for larger data packets (> 64kb), offering much faster response times. In extensive testing with a 100 million vectors dataset, NATS showcased lower vector search and query latency compared to RocksMQ.

Milvus is an open-source vector database used to store, index, and efficiently search high-dimensional vector data. It's particularly useful in applications involving similarity searches across unstructured data, such as Generative Chat responses, product recommendations, and more. By using Airbyte, it's straightforward to transfer data from many different sources into Milvus, calculating vector embeddings of texts along the way. The power of embeddings is to be able to search for relevant pieces of information, even if similar concepts are phrased differently. This article demonstrates how to use Zilliz Cloud as a vector store, Airbyte to extract and load the data, OpenAI embedding API to calculate embeddings, and Streamlit to build a smart submission form showing relevant data.

Christy Bergman, a new Developer Advocate at Zilliz, shares her journey of discovering and choosing Milvus, the world's most popular open-source vector database. She explains how she explored various vector databases, including FAISS, Qdrant, Chroma, Weaviate, Pinecone, and finally settled on Milvus due to its user-friendly experience, speed in loading vectors and querying, and additional features. Christy also discusses her role at Zilliz and her plans for organizing events, writing blogs, improving documentation, and helping developers learn how to use Milvus.

This blog post discusses the integration of NVIDIA Merlin, an open-source framework developed for training end-to-end models to make recommendations at any scale, with Milvus, an efficient vector database created by Zilliz. The integration is beneficial in the item retrieval stage with a highly efficient top-k vector embedding search. The post also highlights how Milvus complements Merlin in recommender systems workflows and provides benchmark results showing impressive speedups with GPU-accelerated Milvus that uses NVIDIA RAFT with the vector embeddings generated by Merlin Models.

Vector embeddings are crucial when working with semantic similarity. They represent input data as a series of numbers, allowing mathematical operations to be performed on the data instead of relying on qualitative comparisons. The appropriate vector embeddings must be obtained before use, as using an image model for text or vice versa may result in poor results. Vector embeddings are influential for many tasks, particularly semantic search. Vector embeddings are created by removing the last layer and taking the output from the second-to-last layer of a deep learning model (embedding models or a deep neural network). The dimensionality of a vector embedding is equivalent to the size of the second-to-last layer in the model. Common vector dimensionalities include 384, 768, 1,536, and 2,048. A single dimension in a vector embedding does not mean anything; however, when all dimensions are taken together, they provide the semantic meaning of the input data. The dimensions represent high-level, abstract attributes that depend on the training data and the model itself. Different models generate different embeddings based on their training data and architecture. To obtain proper vector embeddings, identify the type of data you wish to embed (images, text, audio, videos, or multimodal data) and use appropriate open-source embedding models from Hugging Face or PyTorch. For example, ResNet-50 is a popular image recognition model, while MiniLM-L6-v2 and MPNet-Base-V2 are text embedding models. Vector databases like Milvus and Zilliz Cloud are used to store, index, and search across massive datasets of unstructured data through vector embeddings. They employ the Approximate Nearest Neighbor (ANN) algorithm to calculate spatial distances between query vectors and stored vectors in the database.

Milvus is an open-source vector database designed for similarity search, offering robust storage, processing, and retrieval capabilities for billions of vector data with minimal latency. As of September 2023, it has garnered almost 23,000 stars on GitHub and is used by tens of thousands of users across various industries. The latest release introduces new features such as GPU support and MMap storage for increased performance and capacity. To facilitate the migration process from older versions of Milvus (1.x), FAISS, and Elasticsearch 7.0 and beyond to the latest Milvus 2.x versions, a data migration tool called Milvus Migration has been developed. This powerful tool is written in Go and supports multiple interaction modes, including command-line interface (CLI) using the Cobra framework, Restful API with built-in Swagger UI, and integration as a Go module in other tools. Milvus Migration simplifies the migration process through its robust feature set, which includes support for various data sources such as Milvus 1.x to Milvus 2.x, Elasticsearch 7.0 and beyond to Milvus 2.x, and FAISS to Milvus 2.x. It also supports multiple file formats like local files, Amazon S3, Object Storage Service (OSS), Google Cloud Platform (GCP), and flexible Elasticsearch integration for migrating dense_vector type vectors from Elasticsearch as well as other field types such as long, integer, short, boolean, keyword, text, and double. The migration process involves configuring a migration.yaml file with details about the data source, target, and other relevant settings. Users can then execute the migration job using either command-line or Restful API methods. Once completed, users can view the total number of successful rows migrated and perform other collection-related operations using Attu, an all-in-one vector database administration tool. Future plans for Milvus Migration include supporting migration from more data sources like Redis and MongoDB, adding resumable migration capabilities, simplifying migration commands by merging the dump and load processes into one, and expanding support to other mainstream data sources.