June 2024 Summaries

In a recent talk at the Unstructured Data Meetup, Jiang Chen discussed how to build a Retrieval Augmented Generation (RAG) system using Milvus vector database and Snowflake ecosystem with Snowpark Container Service (SPCS). RAG is an advanced information retrieval method that enhances large language models' response quality by providing relevant context from internal knowledge bases. The integration of Milvus, a powerful open-source vector database, with Snowflake allows users to easily interact with the data stored in Snowflake and build sophisticated applications like RAG systems.

Milvus 2.4 introduces multi-vector columns within a single collection, enabling more advanced and flexible data searches by allowing simultaneous queries across multiple vector types and fields. This hybrid search feature supports multimodal search, hybrid sparse and dense search, and hybrid dense and full-text search. The results from each field are integrated and re-ranked using multiple reranking algorithms to deliver more accurate outcomes. Hybrid search is designed to handle complex and multimodal data representations, accommodating diverse facets of information and various types of vector embeddings. It also supports the fusion of multimodal vectors from different unstructured data types such as images, videos, audio, and text files. The latest Milvus releases support hybrid search to meet escalating demands for handling intricate datasets in AI-driven applications.

This blog post discusses how to build intelligent Retrieval Augmented Generation (RAG) applications using LangServe, LangGraph, and Milvus from the LangChain ecosystem. The author guides readers through setting up a FastAPI application, configuring LangServe and LangGraph, and utilizing Milvus for efficient data retrieval. The post also covers building an LLM agent with LangGraph and integrating Milvus for vector storage and retrieval. Key prerequisites include Python 3.9+, Docker, and basic knowledge of FastAPI and Docker.

In April 2024, Jove Zhong, co-founder of Timeplus, delivered a talk on "Processing Streaming Data in Kafka with Timeplus Proton" at the Seattle Unstructured Data Meetup. Timeplus is revolutionizing real-time data handling with its streaming SQL database and real-time analytics platform. The company's core engine, Timeplus Proton, serves as an alternative to platforms like ksqlDB and Apache Flink. It supports diverse data sources, including Apache Kafka, Confluent Cloud, and Redpanda, allowing for real-time insights and alerts. Jove also discussed the integration of streaming data with Large Language Models (LLMs) and vector databases, emphasizing their potential to enhance AI applications. The talk concluded with a look towards the future of streaming SQL and real-time processing, highlighting its growing importance in various industries.

Storia Lab is a suite of APIs designed to integrate advanced image editing functionalities into applications, making it easier for developers to perform nuanced edits while preserving the integrity of original images. The platform offers solutions such as text correction, background removal and replacement, removing unwanted elements, and sketch-to-image conversion. Storia Lab can also be integrated with Milvus, an open-source vector database designed to handle billion-scale vectors efficiently, for advanced multimodal applications like content creation, image search and recommendation, visual content curation, e-commerce, creative design tools, and visual content moderation.

Large Language Models (LLMs) have revolutionized various tasks such as content generation and customer service chatbots, but ensuring their unbiased knowledge of current news is crucial for reliable responses. Multilingual narrative tracking helps achieve this by analyzing how a narrative is reported in terms of volume and sentiment across different languages and countries. This process involves tracking narratives, which are sequences of interconnected events, such as the Barbie Movie Campaign narrative with its diverse coverage and sentiments across various regions and languages. LLMs should be exposed to diverse news across languages, countries, and other demographics to avoid biased reporting. Multilingual narrative tracking can increase transparency and ensure global perspectives and comprehensive event coverage by including voices from different limits and cultural backgrounds.

Large language models (LLMs) can sometimes produce confident but incorrect information, a phenomenon known as hallucination. This issue is significant in industries like law and healthcare where the accuracy of information generated by LLMs is critical. There are two major categories of hallucinations: intrinsic hallucinations and extrinsic hallucinations. Intrinsic hallucinations tend to contrast the source information given to them, while extrinsic hallucinations occur when LLMs generate information that cannot be verified against the provided source data. Hallucinations can have far-reaching societal implications, undermining trust in reliable information sources and contributing to widespread confusion and mistrust among the public. Several methodologies are used to detect LLM hallucinations: self-evaluation, reference-based detection, uncertainty-based detection, and consistency-based detection. Implementing these approaches ensures the responsible deployment of LLMs and other generative AI technologies, maximizing their positive impact on society.

Recent advancements in artificial intelligence have led to the development of large language models (LLMs), revolutionizing natural language processing. These powerful models, such as ChatGPT and Llama, demonstrate superior capabilities in understanding and generating human-like language but are limited by their training data cut-off date. To unlock their full potential, LLM customization is essential. Customization options include Retrieval Augmented Generation (RAG) and fine-tuning methods like supervised fine-tuning and Reinforcement Learning from Human Feedback (RLHF). RAG enhances response quality by injecting relevant contexts alongside the query, while fine-tuning involves training LLMs on specific data domains.

Misconceptions about data insertion speed in Milvus may arise due to users overlooking the detailed process steps involved. When using libraries like LangChain or LlamaIndex, these platforms convert unstructured data into vectors and then insert them into Milvus Lite. The abstraction of this complex process can create an illusion that the data insertion process takes a long time. However, the actual time-consuming step is generating embeddings from unstructured data, which is computationally intensive. In comparison, the average Milvus vector database insert time is only about a tenth of a second. Thus, around 97% of the "Milvus insert" time observed in LangChain or LlamaIndex is spent on embedding generation, while about 3% is spent on the actual database insertion step.

Personalization is crucial in maintaining long-term customer satisfaction and retention for user-centric products like Netflix, Disney, or food delivery apps. AI recommendation engines leverage historical data to provide personalized experiences. Mike Del Balso, CEO of Tecton, discussed using the RAG architecture to improve AI recommendation engine personalization at a recent Unstructured Data Meetup hosted by Zilliz. He highlighted that AI-powered personalization could add $5 trillion in value to global GDP. RAG (Retrieval Augmented Generation) is an effective technique for enhancing the response quality and relevance of large language models (LLMs). It consists of a retriever, which combines an embedding model and a vector database like Milvus or Zilliz Cloud, and a generator, which is the LLM. The RAG pipeline involves transforming all documents into vector embeddings stored in a vector database, converting user queries into vector embeddings, retrieving top candidates from the vector database based on similarity to the query, and generating a coherent response using the query and Top-K candidates. However, traditional RAG systems lack personalized context for users' likes and dislikes. Full-RAG addresses this by adding context in the retrieval pipeline. This involves providing context on candidate locations (e.g., weather, activities) and user preferences (e.g., historical sites, accommodation). Tecton has developed a feature platform to integrate different business data sources for creating personalized contexts at various levels: Base, Batch Context, Batch + Streaming Data Context, and Batch + Streaming data + Real-time Context. RAG is essential in enhancing AI recommendation engines' effectiveness and long-term customer retention. Tecton simplifies building streaming context by providing a Python SDK for coding context definitions and real-time evaluation of data. However, challenges remain, such as managing trade-offs between speed and costs, integrating third-party real-time data sources, and ensuring proper model governance, debugging, and version control.

Milvus v2.4.3 introduces full-string metadata matching, allowing users to match strings using prefix, infix, postfix, or character wildcard searches. This update makes metadata filtering more versatile and powerful. The blog demonstrates how to use this feature with an example using IMDB movie data. It covers connecting to Milvus Lite, transforming movie text into vectors, inserting vectors and metadata into Milvus, and handling user queries by searching for similar data vectors. Additionally, the blog provides resources and further reading on using array fields in Milvus and filtering searches.

GitHub Next introduced Copilot Workspace, a task-oriented development environment that integrates generative AI models into coding environments. The workspace allows developers to brainstorm, plan, build, test, and run code using natural language conversations and prompts. It follows a task-to-code workflow, starting with task creation and flowing into specification, planning, and coding. Steering points are implemented between the given task and suggested code, allowing developers to guide the model when it misunderstands requests or misses edge cases. The workspace is useful for tasks such as fixing bugs and implementing features.

The article discusses Retrieval Augmented Generation (RAG), a technique that optimizes large language models by providing context from the query. It explains how embedding models convert unstructured data into vector embeddings, enabling computers to understand semantics. RAG is particularly useful in reducing hallucinations in generative AI models like ChatGPT. The article also introduces Voyage AI's domain-specific and general-purpose embedding models and rerankers that contribute significantly to search and RAG. Furthermore, it demonstrates how to integrate Zilliz Cloud Pipelines with Voyage AI for streamlined embedding generation and retrieval, using Cohere as the LLM to build a RAG application.

LLMs can be empowered with important new capabilities through agents that use planning, memory, and tools to accomplish tasks. This post demonstrates how to build agents capable of tool-calling using LangGraph with Llama 3 and Milvus. Agents can perform actions such as web searching, browsing emails, correcting RAGs, and more. The process involves setting up LangGraph, Ollama & Llama 3, and Milvus Lite. Using these tools, a custom local Llama 3 powered RAG agent is built with different approaches like routing, fallback, and self-correction. Examples of agents include the Hallucination Grader and the Answer Grader. The post concludes by compiling the LangGraph graph and testing it.

Large language models (LLMs) have revolutionized AI, particularly in conversational AI and text generation. However, a critical issue that needs to be addressed is the occurrence of logical fallacies in LLM output. Logical fallacies can lead to flawed reasoning and misinformation. There are multiple reasons why these fallacies occur, including imperfect training data, small context window, and the probabilistic nature of LLMs. To tackle this problem, strategies such as human feedback, reinforcement learning, prompt engineering, and more have been proposed. One interesting approach is RLAIF (Reinforcement Learning from AI Feedback), which uses AI to fix itself by detecting and correcting logical fallacies. The FallacyChain module in LangChain has been developed to implement this approach, making LLM outputs more reliable and trustworthy.

Bad data can significantly impact AI-powered applications and workflows, leading to inaccurate results and frustrated users. To address this issue, Voxel51 has developed a solution that brings transparency and clarity to visual AI workflows, making it faster and more efficient to build high-quality datasets and models. By integrating vector databases with tools like Voxel51's FiftyOne open source project, users can test and assess models by feeding them the exact datasets they need for robust, accurate results. This approach accelerates the path to success in AI development, as better data leads to better models. Vector search capabilities are essential in computer vision, offering a powerful engine for data exploration, model evaluation, and innovative multimodal search using embeddings, concept interpolation, and traversal. As AI continues to evolve, integrating vector databases will play a crucial role in shaping the future of unstructured data-driven technologies.

Delivery Hero, a multinational online food delivery company, has implemented an AI safety system to generate high-quality images of products. The system consists of two stages: food image generation and building a safety system. For the first stage, they use DALL-E from OpenAI and implement an image inpainting method with Grounding DINO and DALL-E. In the second stage, four components are used to generate a final score for each image: image tagging, image centering, text detection, and image sharpness. The scores obtained from these components are combined with a weighted function to give each image one final score value. By applying a threshold, an image with a final score below the threshold will be filtered out and not recommended to vendors.

Bo Wang from Jina AI discussed the development of state-of-the-art text embeddings, which power modern vector search and Retrieval-Augmented Generation (RAG) systems. The release of Jina-Embeddings-V2 garnered significant attention in the AI community, with over 3 million downloads on Hugging Face. It has been integrated into various AI frameworks like LangChain and LlamaIndex, as well as vector databases such as Milvus and Zilliz Cloud. Jina embeddings closely compete with OpenAI embeddings. Jina AI initially began by fine-tuning existing models like BERT but soon realized that the industry was not ready for fine-tuning techniques. This led them to develop their own embedding model from scratch, resulting in Jina-Embeddings-V1 and later V2. The latest version, V2, can handle sequences up to 8,192 tokens during inference while training on shorter sequences. Jina-Embeddings-V2 removes position embeddings and introduces Attention with Linear Biases (ALiBi) for dynamic context modeling. It also adapts ALiBi for bidirectional transformers and retrains BERT from scratch, resulting in JinaBERT as the backbone for V2. The model has been successful in handling multilingual data and consistently outperforms competitors like Multilingual E5 and Cohere Embed V3. When developing RAG applications using Jina-Embeddings-V2, it's essential to consider document length and the positioning of relevant information within these documents. The team at Jina AI is already working on Jina-Embeddings-V3, which promises improvements in speed, efficiency, multilingual support, real-world problem solving, task-specific enhancements, and chunk and schema awareness.

During a meetup in May, Charles Packer discussed how MemoryGPT (MemGPT) aims to solve the problem of limited memory in large language models (LLMs). MemGPT introduces a virtually extended context window inspired by computer system design. It divides the LLM context into two parts: main context and external context. The main context has a limited bandwidth, while the external context is stored on persistent storage with an infinite window. MemGPT efficiently manages information flow between the two contexts, allowing for long-context memory applications like personal assistant chatbots.

In March 2024, AJ Steers discussed utilizing Airbyte and PyAirbyte to integrate structured and unstructured data from various sources across different platforms at the SF Unstructured Data Meetup. AJ Steers is an experienced architect, data engineer, software developer, and data ops expert who has designed end-to-end solutions at Amazon and created a vision for quantified self-data models. He currently works as a staff software engineer at Airbyte. Airbyte's focus so far has been on offering reliability, flexible deployment options, and a robust library of connectors to ensure seamless data integration for traditional tabular data. However, the platform has expanded its capabilities in recent months to cover unstructured data sources as well. This expansion includes support for vector database destinations like Milvus, ensuring effective utilization of data across various applications. PyAirbyte is a Python library that provides an interface to interact with Airbyte and allows users to control and manage their Airbyte instances using Python. It offers several advantages, such as the ability to run anywhere, reduce time to value, fast prototyping, and flexibility. Users can choose between the hosted version of Airbyte (no-code approach) or PyAirbyte (minimal code approach) for integrating data sources with data destinations. In conclusion, whether you prefer a no-code or minimal-code approach, Airbyte and PyAirbyte offer robust solutions for integrating both structured and unstructured data from various sources across different platforms.

Milvus Lite is a new open-source vector database that has become the default method for third-party connectors like LangChain and LlamaIndex to connect to Milvus. Comparing timings using the same HuggingFace embedding model, it was found that using Milvus Lite APIs directly provides the best balance between high control over Milvus settings and fast setup. The full code and timings are available on GitHub. This article covers connecting to Milvus Lite using LlamaIndex, LangChain, and Milvus Lite APIs, as well as choosing the right Milvus Light method based on control and speed trade-offs.

The article discusses the role of CPUs in vector search operations and whether they are sufficient to meet modern demands. It highlights a recent talk by George Williams, who explored how new hardware solutions could revolutionize vector search technology. The NeurIPS BigANN competitions aim to push the boundaries of vector search technology, with Zilliz being one of the winners. Zilliz's approach involves optimizing memory layout and access patterns for SSDs, as well as focusing on memory efficiency and maximizing data retrieval speed. The future of vector search may shift from relying on approximation techniques to leveraging brute force methods while still maintaining short timeframes.

Zilliz Cloud, a fully managed version of the open-source vector database Milvus, is now available in 11 regions across three major cloud providers: AWS, Azure, and Google Cloud Platform (GCP). This expansion allows users to deploy Zilliz Cloud closer to their user base, reducing latency and improving performance. The availability includes five regions on AWS, four on GCP, and two on Azure, covering key areas in the Americas, EMEA, and APAC. By offering broader coverage than other vector database vendors, Zilliz Cloud provides more deployment options and ensures better service availability. This expansion reflects Zilliz's commitment to providing the best possible infrastructure for development needs, helping users reduce complexity and lower total cost of ownership (TCO).

Laurie Voss, VP of Developer Relations at LlamaIndex, discussed building advanced Retrieval Augmented Generation (RAG) apps with LlamaIndex in a recent Unstructured Data Meetup. RAG is designed to overcome the limitations of Language Models (LLMs) by assisting them with retrieval capabilities. The main drawback of LLMs is their limited context windows, which can only handle part of an organization's data simultaneously. LlamaIndex is an open-source framework that connects your data to LLMs and simplifies the creation of RAG applications, allowing developers to build functional RAG systems with minimal code. It provides advanced data ingestion and querying features for RAG applications, such as Data Connectors, PDF Parsing, Embedding Models, Vector Stores, Sub-Question Query Engine, Small to Big Retrieval, Metadata Filtering, Hybrid Search, and Agents.

Joan Kusuma's innovative approach to enhancing the fashion retail experience involves using image-based recommendations. By utilizing convolutional neural networks (CNNs) and visual embeddings, she has created a personalized outfit recommendation system that can transform the fashion industry. The process includes image preprocessing, feature extraction, vector search with vector databases, indexing, and image recommendation model in action. Joan's work demonstrates the potential of AI in fashion retail by delivering personalized outfit suggestions using visual embeddings and vector databases.

The text discusses the challenges faced by machine learning teams in deploying large language models (LLMs) into production, such as addressing hallucinations and ensuring responsible deployment. It highlights strategies for conducting quick and accurate LLM evaluations shared by Hakan Tekgul, an ML Solutions Architect at Arize AI, during a recent Unstructured Data Meetup. The article emphasizes the importance of leveraging evaluation tools for seamless LLM observability and explores five primary facets of LLM observability: LLM Evaluations, Spans and Traces, Prompt Engineering, Search and Retrieval, and Fine-tuning. It delves into the LLM Evaluation and LLM Spans and Traces categories in more detail to highlight their significance in optimizing LLM observability. The article concludes by reflecting on Hakan Tekgul's talk, emphasizing that deploying LLMs into production is challenging but can be achieved with a robust observability framework.

Large language models (LLMs) have made significant strides in machine learning and natural language processing, but they face a unique issue called AI hallucinations, where incorrect or false information is generated. This can happen due to lack of context, training data issues, overgeneralization, or design limitations. Retrieval Augmented Generation (RAG) is an advanced approach that aims to enhance the accuracy and reliability of AI models by providing relevant, current information related to a user's question. RAG helps ensure that models can access the newest data, like recent news or research, to give better answers and reduce mistakes. Building a Retrieval Augmented Generation (RAG) system involves several complex steps and decisions, including choosing an embedding model, selecting an index structure, chunking, determining keywords or semantic search, and integrating rerankers. RAG's ability to handle trillions of tokens makes it ideal for handling massive, ever-changing datasets. Combining RAG's precision with the adaptability of long-context models could lead to a powerful synergy. Evaluating large language models (LLMs) can be challenging, but one solution is to have LLMs evaluate each other by generating test cases and measuring the model's performance.