May 2023 Summaries

OpenAI's ChatGPT, powered by GPT-3.5, has revolutionized natural language processing (NLP) and sparked interest in large language models (LLMs). As the adoption of LLMs grows across various industries, so does the need for more advanced AI models that can process multimodal data. The tech world is buzzing with anticipation for GPT-4, which promises to be even more powerful by enabling visual inputs. To prepare for this upcoming revolution, Zilliz has introduced GPTCache integrated with Milvus - a game-changing solution that can help businesses save big on multimodal AI. Multimodal AI refers to integrating multiple modes of perception and communication, such as speech, vision, language, and gesture, to create more intelligent and effective AI systems. This approach allows AI models to better understand and interpret human interactions and environments and generate more accurate and nuanced responses. Multimodal AI has applications in various fields, including healthcare, education, entertainment, and transportation. GPTCache is a project developed to optimize response time and reduce expenses for API calls associated with large models. It enables the system to search for potential answers in the cache first before sending a request to a large model. GPTCache speeds up the entire process and helps reduce the costs of running large models. Semantic cache stores and retrieves knowledge representations of concepts. It is designed to store and retrieve semantic information or knowledge in a structured way. Thus, an AI system can better understand and respond to queries or requests. The idea behind a semantic cache is to provide faster access to relevant information by providing precomputed answers to commonly asked questions or queries, which can help improve the performance and efficiency of AI applications. One of the cornerstones of a semantic cache such as GPTCache is the vector database. Specifically, the embedding generator of GPTCache converts data to embeddings for vector storage and semantic search. Storing vectors in a vector database, such as Milvus, not only supports storage for a large data scale but also helps speed up and improve the performance of similarity search. This allows for more efficient retrieval of potential answers from the cache. The Milvus ecosystem provides helpful tools for database monitoring, data migration, and data size estimation. For more straightforward implementation and maintenance of Milvus, there is a cloud-native service Zilliz Cloud. The combination of Milvus with GPTCache offers a powerful solution for enhancing the functionality and performance of multimodal AI applications. Temperature in machine learning has become a valuable tool to balance randomness and coherence and align with the user's or application's specific needs and preferences. The temperature in GPTCache mainly retains the general concept of temperature in machine learning. It is achieved through 3 options in the workflow: 1. Select after evaluation 2. Call model without cache 3. Edit result from cache GPTCache and Milvus represent an exciting and innovative approach to building intelligent multimodal systems. The following examples showcase how GPTCache and Milvus have been implemented in multimodal situations: 1. Text-to-Image: Image Generation 2. Image-to-Text: Image Captioning 3. Audio-to-Text: Speech Transcription With its support for unstructured data, Milvus is an ideal solution for building and scaling multimodal applications. Furthermore, adding more features in GPTCache, such as session management, context awareness, and server support, further enhances the capabilities of multimodal AI. With these advancements, multimodal AI models have more potential uses and scenarios.

The GPTCache project aims to build a semantic cache for storing large language model (LLM) responses, addressing the challenges of increasing costs and slow response times associated with high traffic levels. LangChain is a library that assists in developing applications combining LLMs with other computational or knowledge sources. Before integrating GPTCache, LangChain's cache was based on string matching, including Memory Cache, SQLite Cache, and Redis Cache. The current condition for hitting the cache requires identical questions, which has limited cache utilization rate. Integration of GPTCache significantly improves cache functionality by performing embedding operations to obtain vectors and conducting vector approximation searches in cache storage. This increases the cache hit rate, reduces LLM usage costs, and speeds up response times.

This tutorial explores the use of vector databases in Jupyter Notebooks, particularly Milvus Lite. Vector databases are useful for working with unstructured data like images, text, or video and can help solve problems faced by large language models (LLMs) such as a lack of domain knowledge and up-to-date data. They also power similarity search applications, product recommendations, reverse image search, and semantic text search. The tutorial covers the basics of vector databases, Milvus Lite, and how to use them in Jupyter Notebooks. It provides examples for using a standalone vector database instance like Milvus Standalone and offers resources for understanding vector databases further.

LangChain is a framework that enables the creation of applications using large language models (LLMs) like GPT. It provides functionalities such as token management and context management, allowing users to build with the CVP Framework. The two core LangChain functionalities for LLMs are data-awareness and agency. One primary use case is querying text data, which can be done using documents, vector stores, or GPT interactions. In this tutorial, we covered how to interact with GPT using LangChain and queried a document for semantic meaning using LangChain with a vector store.

Pymilvus is a Python SDK built for Milvus and Zilliz Cloud, offering access to all features provided by Milvus. However, users have faced issues with the complexity of configuration options available in the vector database system. To address this, MilvusClient was introduced as an attempt to simplify the API for most users. It offers functions such as insert_data(), upsert_data(), search_data(), query_data(), get_vectors_by_pk(), delete_by_pk(), add_partition(), and remove_partition(). The main goal of MilvusClient is to provide easy-to-use operations that may not exist or are unoptimized on the Pymilvus side. As Pymilvus improves, these operations can be optimized behind the scenes while maintaining a simple API for users.

This tutorial demonstrates how to use semantic search with a vector database to analyze speeches given by the Biden administration during their first two years in office. The dataset used is "The White House (Speeches and Remarks) 12/10/2022" found on Kaggle. The process involves cleaning the data, setting up a vector database using Milvus Lite, getting vector embeddings from speeches, populating the vector database, and performing semantic searches based on descriptions. Semantic search allows for finding speeches with similar content rather than just matching exact phrases or sentences.

LlamaIndex is a user-friendly, flexible data framework that connects private, customized data sources to large language models (LLMs). It helps address LLMs' lack of domain-specific knowledge by injecting data. The indexes in LlamaIndex include list index, vector store index, tree index, and keyword index. Each index is made up of "nodes" that represent a chunk of text from a document. LlamaIndex can build many types of indexes depending on the task at hand. It offers an efficient way to query large amounts of data for certain keywords or introduces similarity into LLM applications. The Basics of How to Use LlamaIndex section covers loading a text file, querying the vector store index, and saving and loading an index. Projects that can be created with LlamaIndex include chatbots, web apps, and more.

Auto-GPT is an experimental open-source project that combines a GPT language model with other tools to create an AI system capable of working independently without human intervention. It consists of two core parts: an LLM and a command set, which function as its "brain" and "hands" respectively. However, Auto-GPT has limitations in understanding and retaining extensive contextual information due to the token limit of the GPT model it leverages. Integrating Auto-GPT with a vector database like Milvus can enhance its memory and contextual understanding by converting commands and execution results into embeddings and storing them in the vector database. This integration allows for more precise information retrieval, improving the system's ability to generate aligned commands. Despite some limitations, such as unfiltered top-k results and inability to customize the embedding model, Auto-GPT has immense potential when combined with vector databases like Milvus, pushing the boundaries of AI technology and AIGC systems.

The popularity of large language models (LLMs) like ChatGPT has demonstrated their capabilities in generating knowledge and reasoning. However, these LLMs are pre-trained on publicly available data, which may not provide specific answers and results relevant to a business. LlamaIndex is one solution that can augment LLMs with private data by providing a simple, flexible, centralized interface connecting external data and LLMs. In a recent webinar, Jerry Liu, Co-founder and CEO of LlamaIndex, discussed how LlamaIndex could boost LLMs with private data. Two methods to enhance LLMs with private data were presented: fine-tuning and in-context learning. Fine-tuning requires retraining the network with private data but can be costly and lack transparency. In contrast, in-context learning involves pairing a pre-trained model with external knowledge and a retrieval model to add context to the input prompt. LlamaIndex is an open-source tool that provides central data management and query interface for LLM applications. It contains three main components: data connectors for ingesting data from various sources, data indices for structuring data for different use cases, and a query interface for inputting prompts and receiving knowledge-augmented output. LlamaIndex also manages interactions between the language model and private data to provide accurate and desired results. It operates like a black box, taking in detailed query descriptions and providing rich responses that include references and actions. The vector store index is a popular mode of retrieval and synthesis that pairs a vector store with a language model. LlamaIndex provides numerous integrations, including the integration of Milvus and LlamaIndex. Milvus is an open-source vector database capable of handling vast datasets containing millions, billions, or even trillions of vectors. With this integration, Milvus acts as the backend vector store for embeddings and text. LlamaIndex has various use cases, including semantic search, summarization, text to SQL (structured data), synthesis over heterogeneous data, compare/contrast queries, multi-step queries, exploiting temporal relationships, and recency filtering/outdated nodes.

Zilliz Cloud has released an update that introduces six new features and enhancements, aiming to provide a more robust and cost-effective platform with an enhanced user experience. The latest release includes the Pricing Calculator for better cost estimates, improved system resiliency with data backup and restore on GCP, removal of storage quota for optimal user experience, automatic suspension of inactive databases for credit saving, custom timezone support for more accurate timestamps, and collection renaming for easier database management. Other improvements include a better billing interface, renamed CU types, and additional features to assist users in getting started with Zilliz Cloud.