March 2024 Summaries

JSON, or JavaScript Object Notation, is a flexible data format used for storage and transmission. It employs key-value pairs adaptively, making it ideal for NoSQL databases and API results. Milvus Client, a wrapper around the Milvus collection object, uses a flexible JSON "key":value format to allow schema-less data definitions. This makes it faster and less error-prone than defining a full schema upfront. The schema-less schema includes fields for id (str) and vector (str), with the rest of the fields determined flexibly when the data is inserted into Milvus. JSON data can be uploaded directly into Milvus, which also supports metadata filtering on JSON fields and JSON array data types.

Vector databases, designed to store high-dimensional data points, are particularly useful in handling unstructured data such as image recognition, natural language processing, and recommendation systems. The open-source nature of many vector database projects allows for diverse contributions from various individuals and organizations, fostering innovation and transparency. Open source also promotes accessibility, enabling a wider range of projects and innovations. Community collaboration is crucial in the development of vector databases, with knowledge sharing and inclusive participation playing significant roles. Resources such as well-maintained documentation, chat channels, hackathons, meetups, and conferences contribute to fostering a sense of community and driving innovation. Contributing to open-source vector databases involves finding contribution opportunities, engaging with the community, and understanding that contributions are not limited to coding. Success stories include improvements in scalability, performance, usability, and accessibility of vector databases due to open-source contributions and active community engagement. Challenges faced by these projects include managing a high volume and variety of contributions and balancing diverse interests and visions. However, with robust systems for tracking, reviewing, and integrating contributions, as well as transparent decision-making processes, these challenges can be addressed effectively. In conclusion, the open-source model has proven to be a driving force in advancing vector databases, breaking down barriers, and democratizing access to cutting-edge technology. The diverse community of contributors ensures that these tools are continually improving in terms of robustness, efficiency, and versatility.

Milvus 2.4 introduces CAGRA (CUDA Anns GRAph-based), a GPU-based graph index that significantly enhances vector search performance. Leveraging the parallel capabilities of GPUs, CAGRA offers improved efficiency in both small and large batch queries compared to traditional methods like HNSW. Additionally, CAGRA accelerates index building by approximately 10 times. The integration of CAGRA into Milvus marks a significant milestone in overcoming challenges associated with GPU-based vector search algorithms and sets the stage for future advancements in high recall, low latency, cost efficiency, and scalability in vector search.

Milvus 2.4, a significant update in search capabilities for large datasets, has been released. This version accelerates search efficiency and broadens the horizons towards a unified search platform capable of fulfilling diverse search use cases with exceptional speed and precision. Key highlights include support for NVIDIA's CAGRA Index, Multi-vector Search, Grouping Search, beta support for sparse vector embeddings, and other key enhancements. These updates significantly boost Milvus's performance and versatility for complex data operations.

Zilliz Cloud has partnered with Confluent to unlock semantic search for real-time updates powered by Apache Kafka, Apache Flink, and the Milvus vector database. The new cloud-native, serverless Apache Flink service is now available directly alongside cloud-native Apache Kafka on Confluent's fully managed data streaming platform. This integration enables users to easily build high-quality, reusable data streams for real-time GenAI applications. By leveraging Kafka and Flink as a unified platform, teams can connect to data sources across any environment, clean and enrich data streams on the fly, and deliver them in real-time to the Milvus vector database for efficient semantic search or recommendation.

The author discusses the problem of losing valuable content from Meetup events and how similarity search techniques can be used to address this issue. They introduce Milvus, an open-source vector database that excels in managing complex data landscapes, and SentenceTransformers, a Python framework for generating text embeddings. The author demonstrates how to use these tools to create a system that searches for similar content within Meetup descriptions. By using OpenAI GPT-3.5-turbo to summarize the content of Meetups, they aim to improve search results by reducing noise in event descriptions.

Retrieval Augmented Generation (RAG) is an approach to building AI-powered chatbots that answer questions based on data the model has been trained on. However, natural language retrieval accuracy remains low, necessitating experiments to tune RAG parameters before deployment. Large Language Models (LLMs) are increasingly being used as judges for modern RAG evaluation, automating and speeding up evaluation while offering scalability and saving time and cost spent on manual human labeling. Two primary flavors of LLM-as-judge for RAG evaluation include MT-Bench and Ragas, with the latter emphasizing automation and scalability for RAG evaluations. Key data points needed for Ragas evaluation include the question, contexts, answer, and ground truth answer.

The AI-driven car repair assistant project aims to create an interactive and reliable platform for drivers seeking automotive advice and solutions. Using the OpenAI LLM model with Milvus vector database, the product combines user inputs with AI capabilities to provide relevant diagnostic suggestions. This tool hopes to revolutionize car maintenance and repair in today's digital world by refining the process of identifying car issues and broadening access to expert advice.

Zilliz Cloud is now available on Azure Marketplace, following its successful integration into AWS and GCP marketplaces. This expansion simplifies subscription management and billing, allowing smoother integration into developers' existing Azure workflows. Getting started with Zilliz Cloud on Azure Marketplace involves searching for "Zilliz Cloud," subscribing, configuring the project and SaaS details, linking the Azure Marketplace subscription with a Zilliz Cloud account, and setting Azure Marketplace as the payment method. This integration enables developers to easily incorporate Zilliz Cloud's powerful capabilities into their AI projects.

In 2023, large language models (LLMs) gained immense popularity, leading to the development of two main types of LLM applications: retrieval augmented generation (RAG) and AI agents. RAG involves using a vector database like Milvus to inject contextual data, while AI Agents use LLMs to utilize other tools. This article combines these two concepts by building an AI Agent for RAG using Milvus and LlamaIndex. The tech stack includes Milvus, LlamaIndex, and OpenAI (or alternatively OctoAI or HuggingFace). The process involves spinning up Milvus, loading data into it via LlamaIndex, creating query engine tools for the AI Agent, and finally building the AI Agent for RAG. This architecture allows an AI Agent to perform RAG on documents by providing it with the necessary tools for querying a vector database.

Stephen Batifol, Developer Advocate at Zilliz in Berlin, is organizing events and creating content to help people understand and use Milvus. With experience as an Android developer, data scientist, machine learning engineer, and now a developer advocate, he has always aimed to simplify the work of data scientists and software engineers. His interest in open-source projects led him to Zilliz, where he is excited to build a community from scratch and engage with people at various events. Batifol plans to immerse himself technically in the domain, start a new Meetup series in Berlin, and release open-source projects soon. He encourages interested candidates to join Zilliz as Developer Advocates across different regions.

Google's Gemini 1.5, an LLM capable of handling contexts up to 10 million tokens, and OpenAI's Sora, a text-to-video model, have sparked discussions about the future of AI, particularly the role and potential demise of Retrieval Augmented Generation (RAG). Gemini 1.5 Pro supports ultra-long contexts of up to 10 million tokens and multimodal data processing. In a "needle-in-a-haystack" evaluation method, Gemini 1.5 Pro achieves 100% recall from up to 530,000 tokens and maintains over 99.7% recall from up to 1M tokens. Even with a super long document of 10M tokens, the model retains an impressive 99.2% recall rate. While Gemini excels in managing extended contexts, it grapples with persistent challenges encapsulated as the 4Vs: Velocity, Value, Volume, and Variety. LLMs’ 4Vs Challenges include hurdles in achieving sub second response times for extensive contexts, considerable inference costs associated with generating high-quality answers in long contexts, vastness of unstructured data that may not be adequately captured by an LLM, and diverse range of structured data. Strategies for optimizing RAG effectiveness include enhancing long context understanding, utilizing hybrid search for improved search quality, and leveraging advanced technologies to enhance RAG’s performance. The RAG framework is still a linchpin for the sustained success of AI applications. Its provision of long-term memory for LLMs proves indispensable for developers seeking an optimal balance between query quality and cost-effectiveness.

The blog post discusses the use of vector databases, such as Milvus or Zilliz Cloud, as a JSON (or relational) datastore. It explains how to create a collection in Milvus and perform CRUD operations on JSON data stored within it. The author demonstrates querying, updating, and deleting records using Python code snippets. Additionally, the post introduces a package called milvusmongo that implements basic CRUD functionality across collections using Milvus as the underlying database instead of MongoDB. The author emphasizes that vector databases are not meant to replace NoSQL databases or lexical text search engines but can be used as an efficient data store for solo developers and small teams, with the option to optimize infrastructure usage later as they grow.