July 2024 Summaries

The Milvus Multimodal RAG demo is a product recommendation system that uses Google's MagicLens multimodal embedding model to encode both images and text into a single multimodal vector. This vector is then used to search for the closest-matching Amazon products from a Milvus vector database. The technologies used in this demo include Google DeepMind's MagicLens, OpenAI's GPT-4o, Milvus, and Streamlit. The data comes from the Amazon Reviews 2023 dataset, with a subset of 5K items being used for the demonstration. The setup instructions for MagicLens involve setting up an environment, installing dependencies, and downloading model weights. The Milvus server is used to store, index, and search vectors, while Streamlit provides a user-friendly interface for uploading images and entering text instructions. The Ask GPT function utilizes OpenAI's GPT-4o mini multimodal generative model to provide AI-powered recommendations based on the search results.

Function calling with LLMs allows developers to create powerful and context-aware applications by integrating Language Learning Models (LLMs) like Llama 3.1 with external tools such as user-defined functions or APIs. This enables the creation of solutions for data extraction, natural language conversion to API calls or database queries, and conversational knowledge retrieval engines that interact with a knowledge base. In this blog post, we explore how to integrate Llama 3.1 with external tools like Milvus and APIs to build advanced applications. The integration of LLMs with external tools opens up new possibilities for developers to create versatile and powerful AI applications catering to specific use cases and practical problems.

Large language models (LLMs) are increasingly being adopted across various industries and production environments. Ensuring their outputs are accurate, reliable, and unbiased is crucial as they become more widespread. Traditional human evaluation methods often fall short due to their time-consuming nature and inconsistency in handling the complexity and scale of modern LLMs. One promising approach to this challenge is using LLMs as judges to evaluate their outputs. By leveraging their extensive training data and contextual understanding, LLMs can provide automated, scalable, and consistent assessments. During a meetup hosted by Zilliz, Sourabh Agrawal discussed the real-world difficulties of implementing LLM-as-a-judge techniques and highlighted four primary metrics for assessing LLM performance: response quality, context awareness, conversational quality, and safety. He also shared strategies for addressing challenges associated with using LLMs as judges, such as biases in evaluations, consistency problems, lack of domain-specific knowledge, and the complexity of evaluating complex responses. To tackle these limitations, developers can adopt objective evaluations, check for conciseness, use a grading system with "YES, NO, MAYBE" options, and maintain cost-effective evaluations by leveraging cheaper LLMs as much as possible. Additionally, fine-tuning the judge LLM for specific domains ensures more accurate and relevant evaluations. UpTrain AI is an open-source framework that developers can use to evaluate their LLM applications. It provides scores and explanations, breaking down long responses into subparts and evaluating each for a more objective measure of conciseness. The UpTrain dashboard logs all data, enabling comparison of models and prompts and monitoring performance.

Retrieval Augmented Generation (RAG) is a technique that enhances large language models (LLMs) by providing them with additional knowledge and long-term memories through vector databases like Milvus and Zilliz Cloud. While RAG can address many LLM headaches, it may be insufficient for more advanced requirements such as customization or greater control of the retrieved results. Knowledge Graphs (KG) can be incorporated into the RAG pipeline to improve performance and accuracy. By integrating KGs with RAG systems, users can enhance contextual understanding, improve accuracy and factual consistency, enable multi-hop reasoning capabilities, facilitate efficient information retrieval, provide transparent and traceable outputs, synthesize knowledge across domains, and handle ambiguity more effectively.

This blog post provides a detailed guide on how to deploy an open-source vector database called Milvus on Amazon Elastic Kubernetes Service (EKS). The author explains the architecture of Milvus and its integration with EKS, AWS S3 for object storage, Amazon Managed Streaming for Apache Kafka (MSK) for message storage, and Amazon Elastic Load Balancing (ELB) as a load balancer. The post also covers prerequisites such as installing the AWS Command Line Interface (CLI), EKS tools like kubectl, eksctl, and helm, creating an S3 bucket with a KMS customer-managed key, and setting up an Amazon MSK instance. Next, the author guides users through creating an Amazon EKS cluster using eksctl, installing the AWS Load Balancer Controller, and deploying Milvus on EKS using Helm. The post also explains how to configure S3 as object storage, MSK as message storage, expose Milvus services for external access, and enable high availability deployment of Milvus core components. Finally, the author demonstrates how to access and manage Milvus endpoints through Kubernetes Services and Attu, an open-source Milvus administration tool. The post concludes with a test using Milvus' official example code to verify if the Milvus database is working properly.

LangChain is an open-source framework that simplifies building conversational AI agents using large language models (LLMs). It provides tools and templates to create smart, context-aware chatbots and other applications. Conversational agents are software programs that interact with users in natural language, handling tasks like answering questions or translating languages. LangChain Agents use LLMs to interact with external tools and data sources, making them more powerful for various applications. To build a conversational agent using LangChain, developers need to install dependencies such as LangChain, langchain-openai, OpenAI API SDK, dotenv, Milvus, pymilvus, and tiktoken. They can then create a conversation chain with the ConversationChain class from langchain.chains, making predictions by passing user input to the conversation chain. To enhance conversational agents with long-term memory, developers can integrate Milvus Lite as a vector store to store and retrieve data efficiently. By incorporating memory into their agents using LangChain and Milvus Lite, developers can create more accurate and personalized responses based on previous interactions. This integration significantly enhances the capabilities of AI agents, allowing them to provide better assistance in various applications.

Retrieval Augmented Generation (RAG) is a technique that enhances Language Learning Models (LLMs) by integrating additional data sources. Milvus, a vector database, can boost the performance of RAG applications with its Metadata Filtering, Hybrid Search, and Agent capabilities. Metadata Filtering allows for precise and efficient searches by enriching data with additional attributes. Hybrid Search expands search capabilities by allowing queries across multiple vector columns. Agents automate actions based on LLM's output, enabling continuous updates to the RAG system.

In this blog post, we explore how Retrieval Augmented Generation (RAG) can be applied to legal data using Ollama and Milvus. RAG is a technique that enhances Language Learning Models (LLMs) by integrating additional data sources. We demonstrate how to set up a RAG system for legal data, leveraging Milvus as our vector database and Ollama for local LLM operations. The process involves indexing the data, retrieval and generation at runtime, and using an LLM to generate a response based on enriched context. This approach can significantly streamline legal research by making it more efficient and easier.

Building a scalable vector search pipeline in production is challenging due to handling massive amounts of unstructured data and high query volumes. To address this, a combination of Milvus, an open-source vector database, and Apache Spark, a distributed computing framework, can be used. Milvus enables efficient vector search operations on large datasets, while Spark accelerates data processing tasks by distributing them across multiple computers in batches. By integrating these tools, developers can create production-ready applications that leverage AI models for improved information retrieval and search processes.

Darren Oberst, CEO of AI Blocks, discussed deploying Retrieval Augmented Generation (RAG) on-premises for large financial and legal services companies during a recent Unstructured Data Meetup session. He highlighted the challenges faced by enterprises in adopting RAG, including data privacy and security concerns, elevated costs, and neglecting retrieval strategies. To address these issues, Darren advocates for deploying RAG on private cloud solutions, offering better data security, lower cost, and enhanced generation with retrieval capabilities. The session also covered the Dragon models designed specifically for Retrieval Augmented Generation (RAG) in the Huggingface Transformers library and LLMware, a library designed for enterprise-level LLM-based applications.

The talk by Jacob Marks at the Unstructured Data Meetup hosted by Zilliz focused on building robust multimodal Retrieval Augmented Generation (RAG) pipelines using FiftyOne, LlamaIndex, and Milvus. RAG enhances large language models' capabilities by augmenting their knowledge with relevant external data. The architecture of a text-based RAG system is simple, integrating LLMs with vector databases like Milvus or Zilliz Cloud to provide users with more accurate and contextually relevant responses. Multimodal RAG proves invaluable for systems that need multiple data types to make informed decisions. It combines information retrieval and generative modeling to enhance the capabilities of multimodal LLMs, integrating various data types such as text, images, audio, and video. The fiftyone-multimodal-rag-plugin can be used to implement a multimodal RAG pipeline using FiftyOne, LlamaIndex, and Milvus.

OpenAI's ChatGPT has sparked a surge of interest in large language models (LLMs) among corporations, leading to increased demand for technology vendors that support LLM operations (LLMOps). These vendors provide comprehensive workflows for developing, fine-tuning, and deploying LLMs into production environments. Sage Elliott, a machine learning engineer at Union.ai, discussed deploying and managing LLMs during a recent Unstructured Data Meetup, focusing on ensuring the reliability and scalability of LLM applications in production settings. LLMOps stands for Large Language Model Operations, which are analogous to MLOps but specifically for large language models (LLMs). MLOps (Machine Learning Operations) refers to the practices and tools used to efficiently deploy and maintain machine learning models in production environments. It is an extension of DevOps (Development and Operations), which integrates application development and operations into a cohesive process. Continuous Integration/Continuous Deployment (CI/CD) is one of the core principles of LLMOps, automating the LLM application development lifecycle. Continuous integration (CI) involves automatically taking application updates and merging them with the main branch, while continuous delivery/deployment (CD) refers to the process of automatically deploying changes to the application in a production environment after integration and validation. LLMOps is essential for production-level AI applications, with the exact infrastructure dependent on the application's needs. Integrating LLMOps into your AI application offers benefits such as resource management and scalability, model updating and improvements, and ethical and responsible AI practices. A simplified LLMOps pipeline includes elements like Sys Prompt, Model, Guardrail, Data Store, Monitor, and CI/CD Orchestrator. The HuggingFace Spaces platform streamlines the shipping of your model into production, offering low-cost cloud GPUs to power LLMs. To get started with LLMOps, follow a simple three-step philosophy: ship the model, monitor its performance, and improve it based on insights gained from monitoring. Tools like LangKit, Ragas, Continuous Eval, TruLens-Eval, LlamaIndex, Phoenix, DeepEval, LangSmith, and OpenAI Evals can help evaluate LLM applications.

Retrieval Augmented Generation (RAG) systems have significantly enhanced AI applications by providing more accurate and contextually relevant responses. However, scaling and deploying these systems in production have presented considerable challenges as they become more sophisticated and incorporate custom AI models. BentoML is a valuable tool that simplifies the process of building and deploying inference APIs for custom models, optimizes serving performance, and enables seamless scaling. By integrating BentoML with the Milvus vector database, organizations can build more powerful, scalable RAG systems.

The introduction of specialized GenAI frameworks like LangChain has enabled developers to build sophisticated AI applications quickly and easily by leveraging powerful large language models (LLMs) such as ChatGPT and LLaMA. These frameworks allow users to create Retrieval Augmented Generation (RAG) applications in just a few lines of code without requiring deep theoretical AI knowledge. However, these GenAI frameworks are typically written in Python, which may not be familiar to all full-stack engineers and software developers. There is a need for extensions of these GenAI frameworks in other programming languages so that these full-stack engineers can leverage powerful LLMs to build GenAI applications in their software projects. Andrei Bondarev introduced a Ruby extension of LangChain called LangChain.rb to make it easier for full-stack engineers to build GenAI applications in their software projects.

Jithin James and Shahul Es shared insights on leveraging metrics-driven development to evaluate Retrieval Augmented Generation (RAG) systems at the Zilliz Unstructured Data Meetup. They discussed both the theoretical foundations and practical applications of RAG system evaluation, explaining how understanding the theory behind the evaluation code can provide deeper insights into its functionality. The talk covered key metrics such as factual similarity and semantic similarity to determine the quality and relevance of the generated answer compared to the ground truth. Practical examples were provided on how to evaluate and improve a RAG system powered by Milvus, an open-source vector database known for its efficiency in similarity search and AI applications.

Retrieval Augmented Generation (RAG) is a technique that uses an AI chatbot with personal data. Three key strategies to optimize RAG include smart text chunking, iterating on different embedding models, and experimenting with various LLMs or generative models. Smart text chunking involves breaking down text into manageable pieces for efficient retrieval by the Vector Database. Different techniques for this process include recursive character text splitting, small-to-big text splitting, and semantic text splitting. Iterating on embedding models determines how data is represented as vectors, which are crucial in AI applications. Lastly, experimenting with different LLMs allows users to choose the most suitable one for their workload.