December 2023 Summaries

Optimizing Retrieval Augmented Generation (RAG) applications involves using methodologies, metrics, and evaluation tools to enhance their reliability. Three categories of metrics are used in RAG evaluations: those based on the ground truth, those without the ground truth, and those based on LLM responses. Ground truth metrics involve comparing RAG responses with established answers, while metrics without ground truth focus on evaluating the relevance between queries, context, and responses. Metrics based on LLM responses consider factors such as friendliness, harmfulness, and conciseness. Evaluation tools like Ragas, LlamaIndex, TruLens-Eval, and Phoenix can help assess RAG applications' performance and capabilities.

Picdmo, an AI-powered photo management app, sought to improve its search performance and user experience. The team initially used Milvus, an open-source vector database, but found it labor-intensive and financially burdensome. They then integrated Zilliz Cloud, a fully managed Milvus service, into their infrastructure. This resulted in response times plummeting from 8 seconds to less than 1 second, even under extreme data loads. The adoption of Zilliz Cloud brought efficient search performance, substantial time and cost savings, and responsive support from the Zilliz team. As Picdmo evolves into a comprehensive multimedia application, its collaboration with Zilliz remains crucial for future features.

In the data-driven world, the exponential growth of unstructured data has led to the rise of vector databases. These powerful tools specialize in storing, indexing, and searching unstructured data through high-dimensional numerical representations known as vector embeddings. They are used for building recommender systems, chatbots, and applications for searching similar images, videos, and audio. When selecting a vector database, scalability, functionality, and performance are the top three most crucial metrics to consider. Scalability is essential for accommodating growing data demands effectively, while functionality includes both vector-oriented features like support for multiple index types and database-oriented features such as Change Data Capture (CDC) and multi-tenancy support. Performance is evaluated using benchmarking tools like ANN-Benchmark and VectorDBBench, which measure recall rate, QPS, latency, and other metrics. Various vector search technologies are available beyond vector databases, including vector search libraries, lightweight vector databases, vector search plugins, and purpose-built vector databases. Each type has its strengths and weaknesses, so the choice depends on specific business needs.

This post discusses database schemas, specifically focusing on vector databases and their dynamic schema feature. It explains that SQL databases have predefined schemas while NoSQL databases typically have a dynamic or schemaless schema. The Milvus vector database supports dynamic schema, allowing users to add data in JSON format without defining attributes when creating the database. The article covers how to use dynamic schema with the Milvus vector database and how the feature is implemented. It also discusses the pros and cons of dynamic schemas, such as ease of setup and flexibility but slower filtered search compared to fixed schemas.

Trend Micro, a global leader in cybersecurity, has integrated Milvus, an open-source vector database, into their security infrastructure to enhance APK (Android application package) security. The company initially used MySQL for APK similarity search but faced scalability issues as the dataset grew. They then shifted focus to Faiss, which excelled in speed but lacked critical features required for a production environment. Milvus addressed these challenges with seamless integration with mainstream vector index libraries and simple, intuitive APIs. The implementation of Milvus has resulted in low query latency and high data import speed, significantly enhancing Trend Micro's ability to detect and neutralize harmful APKs.

The text discusses the use of vector databases like Milvus and Zilliz Cloud, which allow hybrid vector and scalar searches. It explains how metadata filtering can be used to perform more precise results that cater to specific needs by limiting search with certain conditions using boolean expressions on scalar fields or primary key field. The text also provides a step-by-step guide on how to create collections and pipelines in Zilliz Cloud, as well as searching via the web console or API calls.

BIGO, the owner of short video platform Likee, has leveraged Milvus, an open-source vector database, to optimize its duplicate video removal process. With millions of daily uploads on Likee, the proliferation of duplicate videos posed a threat to content quality and user experience. Previously, BIGO used FAISS for similarity search but faced limitations in managing massive vectors. Milvus provided faster query responses and scalability, improving throughput and efficiency. The transformation involved converting new video frames into feature vectors and matching them against an extensive database of existing content using cutting-edge technologies like Kafka, deep learning models, and relational databases. BIGO plans to extend Milvus's capabilities for content moderation, restriction, and customized video services in the future.

Prompt engineering techniques are being used to help large language models (LLMs) handle pronouns and other complex coreferences in retrieval augmented generation (RAG) systems. RAG combines the power of LLMs with a vector database acting as long-term memory, enhancing the accuracy of generated responses. One example is Akcio, an open source project that offers a robust question-answer system. However, implementing RAG systems introduces challenges, particularly in multi-turn conversations involving coreference resolution. Researchers are turning to LLMs like ChatGPT for coreference resolution tasks, but they occasionally produce direct answers instead of following the prompt instructions. A refined approach using few-shot prompts and Chain of Thought (CoT) methods has been developed to guide ChatGPT through coreference resolution, resulting in coherent responses.

This article discusses vector similarity search metrics and how they work. It covers three primary distance metrics: L2 or Euclidean distance, cosine similarity, and inner product. Additionally, it mentions other interesting vector similarity or distance metrics such as Hamming Distance and Jaccard Index. The article explains the concept of vectors in terms of orientation and magnitude, and how these metrics can be used to compare any data that can be vectorized. It also provides examples of when each metric should be used.

In order to stay competitive in the e-commerce industry, Shopee ventured into short video services. However, they faced challenges handling vast amounts of unstructured data such as videos, images, audio, and text. Milvus emerged as a solution due to its ability to handle billions of vectors, scalability, and seamless integration with Shopee's internal ecosystem. The migration from Milvus 1.x to 2.x improved stability, scalability, and multi-replica capabilities, resulting in low-latency and high-availability retrieval services. With Milvus, Shopee has elevated its real-time search capabilities and streamlined offline data retrieval for copyright video matching and video deduplication processes.

Zilliz has introduced its new service, Zilliz Cloud Pipelines, which simplifies the process of creating and retrieving unstructured data as vectors. This solution is designed to empower developers in building high-quality semantic searches without requiring extensive customization or infrastructure adjustments. The platform consists of three specific pipelines: Ingestion, Search, and Deletion. Zilliz Cloud Pipelines currently focuses on semantic search in text documents but will be expanded to include image search, video copy detection, and multi-modal search capabilities in the future.

This article explains how to build a movie recommender system using the open source vector database, Milvus. The process involves setting up the environment, collecting and preprocessing data, connecting to Milvus, generating embeddings for movies, sending embeddings to Milvus, and finally recommending new movies using Milvus. By leveraging vector storage and similarity search, Milvus can help build an efficient and scalable movie recommendation system, enhancing user engagement and showcasing the role of advanced vector-based models in modern recommendation systems.