May 2024 Summaries

LlamaIndex has introduced the Property Graph Index to enhance the flexibility, extensibility, and robustness of its knowledge graph capabilities. This new feature addresses the limitations of traditional knowledge graph representations, like knowledge triples, by using a labeled property graph representation, allowing for richer modeling, storage, and querying. The Property Graph Index supports categorizing nodes and relationships with associated metadata and enables complex queries using the Cypher graph query language. Users can combine several extraction methods—schema-guided, implicit, and free-form—to build knowledge graphs, while also utilizing various querying techniques such as keyword-based retrieval, vector similarity, and custom graph traversal. The Property Graph Index operates with a PropertyGraphStore abstraction for data storage and retrieval and is compatible with several backing stores, including Neo4j. This development is in collaboration with Neo4j, and users are encouraged to engage with the LlamaIndex community for support and to share their projects.

The integration of LlamaIndex with PostgresML, a machine learning platform based on PostgreSQL, offers a streamlined solution for Retrieval-Augmented Generation (RAG) workflows by consolidating document storage, splitting, embedding, and retrieval into a single system. This integration addresses common issues in typical RAG workflows, such as performance latency, privacy concerns, and the complexity of managing multiple services, by reducing the need for multiple network calls and leveraging an in-database approach. The PostgresML Managed Index enhances user experience with faster, more reliable, and cost-effective operations while maintaining transparency and flexibility through open-source models. It provides tools for model serving, storing, training, and feature access, enabling efficient AI applications. The integration allows users to execute advanced ML/AI tasks with seamless scalability and reduced latency, significantly improving the speed and reliability of RAG workflows.

LlamaIndex's latest weekly update showcases a range of integration updates, guides, demos, tutorials, and webinars designed to enhance user capabilities in handling data and developing applications. Key highlights include the secure execution of LLM-generated code using AzureCodeInterpreterTool integrated with Azure Container Apps, and the creation of automated email agents with MultiOn and LlamaIndex for efficient email management. LlamaFS, developed by Alex Reibman's team, offers improved file organization capabilities, and RAGApp provides a no-code interface for deploying chatbots. Recent feature releases include dynamic sessions on Azure for secure code execution, local operation of Nomic embed for optimized embeddings, Vespa vector store integration for hybrid search, and the facilitation of batch data processing with MyMagic AI. The update also includes various guides and tutorials, such as building an automated email agent, a full-stack job search assistant, and a RAG chatbot, alongside a webinar on constructing an open-source coding assistant using OpenDevin.

MultiOn is an AI agents platform that enables developers to automate tasks within web environments, while LlamaIndex serves as an orchestration framework to bridge private and public data for applications using Large Language Models. The integration of MultiOn and LlamaIndex allows for seamless automation of web interactions, exemplified through a technical walkthrough that demonstrates managing email interactions and web browsing. This integration involves setting up AI agents with specific configurations, using tools like Gmail search to retrieve and analyze emails, and executing tasks such as email summarization and response generation. The process culminates in utilizing the MultiOn agent to send emails through a web browser, showcasing the practical application of these technologies in streamlining online tasks. As the capabilities of MultiOn and LlamaIndex continue to expand, they are poised to significantly enhance the efficiency of AI-driven digital interactions.

MyMagic AI offers an advanced API for batch inference, enabling users to process and analyze large datasets using various open-source Large Language Models (LLMs) like Llama 70B and Mistral 7B for tasks including data extraction, summarization, categorization, and sentiment analysis. Integrated with LlamaIndex, this system facilitates efficient data management by allowing users to organize data in cloud storage and execute API requests for batch processing tasks, which can be both synchronous and asynchronous. Use cases for the API include information extraction, classification, embedding text files for machine learning, generating training data, and transcription, with integration into LlamaIndex's RAG pipeline supporting effective data storage and retrieval. The API enhances workflow flexibility by enabling dynamic experimentation with different models and prompts, although it currently lacks real-time interactive capabilities, which are planned for future updates.

Leveraging the capability of large language models (LLMs) to generate executable code, the Azure Code Interpreter tool allows for sandboxed code execution, addressing the inherent risks of executing potentially flawed or dangerous code by using Azure Container Apps. This tool, integrated with LlamaIndex, facilitates complex tasks by isolating the code from the host environment, ensuring safe execution. Examples include determining the current time in Seattle, analyzing CSV data, and modifying files, showcasing the tool's ability to handle tasks that involve data inspection and manipulation. Despite an initial syntax error during a file modification task, the tool demonstrated adaptability by correcting and successfully executing the code, highlighting its potential to expand the scope of tasks that LLM agents can perform safely. This innovation promises to enhance the versatility and reliability of LLM-based agents, encouraging users to explore new possibilities with the confidence of secure execution.

Last week's updates from LlamaIndex highlighted significant advancements in AI, particularly with the integration of GPT-4o and Gemini models. Key developments include the launch of LlamaIndex on Vertex AI, featuring a new RAG API that facilitates flexible data source connections and supports the latest LLMs. This period also saw the introduction of GPT-4o in LlamaParse for superior document parsing, albeit at a higher cost. Additionally, a new cookbook was released for structured image extraction using GPT-4o, promising high-quality results. The updates also encompassed guides and tutorials on enhancing QA with LlamaParse, speeding up vector searches using Jina Embeddings, and setting up local research assistants. The community is invited to join a webinar on long-term memory for autonomous agents and to attend a meetup in San Francisco to discuss the latest in generative AI.

LlamaIndex has announced a new collaboration with DeepLearningAI to offer a course titled "Building Agentic RAG," which teaches how to create a research assistant capable of reasoning over multiple documents and integrating human feedback. Recent updates from LlamaIndex include the release of day 0 support for GPT-4o in Python and TypeScript, along with demo notebooks to facilitate experimentation. They have also launched Llama3 cookbooks for various use cases, from chat functionalities to agent development. The latest guides cover building agents in TypeScript and using RAG for content moderation, while tutorials explore advanced applications like extracting elements from complex PDFs with LlamaParse, generating PowerPoint presentations with Llama3, and developing financial agents for stock analysis. Additionally, a tutorial on evaluating RAG systems with tools like TruLens and Ragas focuses on assessing metrics such as faithfulness and relevance.

Llamafile, an open-source project by Mozilla, provides a straightforward method to execute large language models (LLMs) locally on a laptop without significant installation requirements. By downloading a llamafile from platforms like HuggingFace, users can run LLMs directly on their computers, offering benefits such as enhanced privacy, high availability without internet dependence, and the ability to test various models cost-free. This guide demonstrates setting up a llamafile and integrating it with LlamaIndex to create a local research assistant for studying homing pigeons. The process involves downloading a llamafile, making it executable, and using it to index data and generate embeddings, enabling a fully local and private data interaction experience. This setup not only avoids third-party data sharing but also allows experimentation with different models and topics.

The LlamaIndex community is hosting its first hackathon in San Francisco, and although not everyone may be able to attend, the event promises exciting project developments that will be shared afterward. The latest updates include significant releases for LlamaIndex.TS, which has reached version 0.3 with features like agent support and enhanced deployment capabilities, and LlamaIndex Python, now at version 0.10.34, introducing introspective agents and support for various AI tools and platforms. Demonstrations highlight applications such as filtering Airbnb listings using natural language and setting up a local, open-source RAG pipeline. Guides and tutorials offer insights into building complex financial calculation agents, improving RAG accuracy through reranking, and implementing agentic RAG with production-ready techniques. Upcoming webinars will further explore deploying AI applications and advancing RAG pipelines.