March 2024 Summaries

LangFriend is a journaling app developed by LangChain to explore the concept of memory in large language models (LLMs), aiming to improve user experience by leveraging long-term memory for personalized interactions. The app draws inspiration from academic research, such as MemGPT and Generative Agents, which incorporate memory functions into LLMs, and from companies like Plastic Labs and Good AI, which are integrating long-term memory into AI systems. LangFriend’s journaling format is chosen to capture meaningful insights and feelings, allowing the app to remember and use this information to tailor responses. Users can customize the app's system prompts, enhancing its adaptability. The app serves as a research preview, encouraging community feedback to refine and expand its capabilities, ultimately showcasing the potential for memory integration to enhance generative AI applications.

A new service has been launched to demonstrate the potential of extracting structured data from unstructured sources like text and PDF documents, utilizing large language models. This hosted application, although not for production use, allows developers to experiment with the latest tools to create their own data extraction workflows. It supports various file types and models, enabling the definition of extraction schemas in natural language and the sharing of extractors among users. The service relies on features such as long context windows, function calling capabilities, and improved support for handling different file types and schema formats. An example of its application includes extracting financial data from a public company's earnings call transcript, showcasing how structured data can be derived from natural language text. This service is integrated with LangChain's tools and can be connected to larger workflows, providing developers with a powerful tool to explore data extraction possibilities.

"Self Learning GPTs" is a demo app designed to demonstrate how applications can improve over time through feedback collection and utilization. Utilizing LangSmith, the app captures feedback and creates few-shot examples to enhance prompts, thereby facilitating automatic learning and improvement. Learning from feedback has long been a critical component in developing LLM systems, as evidenced by models like ChatGPT and Midjourney, which utilize feedback loops to optimize performance. LangSmith emphasizes feedback collection by allowing users to programmatically log feedback and attach it to specific runs, creating datasets that can be used to improve applications. The app employs LangSmith's features to trace runs, capture feedback, and curate datasets of positive examples, which can then be used to enhance LLM prompts. This process is automated through LangSmith Automations, which will soon be widely available. The initiative not only simplifies dataset construction but also provides practical ways to apply feedback to improve applications. While the current system does not optimize example selection, it lays the groundwork for more advanced optimization using frameworks like DSPy, which focuses on algorithmically optimizing LLM prompts and weights. LangSmith and DSPy share a focus on tracing as a vital component in optimizing LLM systems, allowing for the association of high-level feedback with specific LLM calls. The developers aim to explore further optimization strategies in collaboration with experts like Omar Khattab, who will participate in an upcoming webinar on LLM system optimization.

OpenAI's launch of ChatGPT marked the start of the generative AI era, leading to widespread adoption across various industries. As enterprises shift from prototyping to productionizing large language model applications, they increasingly prefer self-hosted solutions over third-party services. To address this need, LangChain has integrated NVIDIA NIM, a set of microservices designed to streamline the deployment of generative AI at scale. NVIDIA NIM supports various AI models and is built on robust inference engines, allowing enterprises to deploy AI applications confidently, both on-premises and in the cloud. Notably, the self-hosted nature of NIM ensures data privacy, particularly beneficial for applications handling sensitive information. LangChain offers an integration package for using NIM, facilitating the development of AI applications like retrieval-augmented generation (RAG) systems. The integration also supports advanced retrieval methods, such as hypothetical document embeddings, to enhance search query accuracy. This capability is demonstrated through a step-by-step example using LangSmith documentation, showcasing how to build a RAG application with NVIDIA NIM and LangChain frameworks.

Tomaz Bratanic's guest blog post details the process of constructing and retrieving information from knowledge graphs in Retrieval-Augmented Generation (RAG) applications using Neo4j and LangChain. It highlights the benefits of Graph RAG, which utilizes graph databases to organize data as nodes and relationships, enhancing the depth and contextual understanding of retrieved information compared to traditional vector search methods. The blog explains the challenges of constructing a knowledge graph and introduces the LLMGraphTransformer module, which leverages large language models (LLMs) to automate significant parts of this process. It demonstrates setting up a Neo4j database and using LangChain to ingest data, transform it into a knowledge graph, and store it in Neo4j. The post further explores a hybrid retrieval approach combining graph, vector, and keyword searches to enhance the accuracy and depth of RAG applications, culminating in the creation of a comprehensive retrieval chain that integrates structured and unstructured data to generate precise responses. The practical application of this method is illustrated through querying historical data about Elizabeth I, demonstrating the seamless integration of structured graph data with unstructured text retrieval to generate detailed and accurate answers.

The text explores various approaches to handling high-cardinality categorical values in large language models (LLMs), focusing on use cases where structured data output is required, such as query analysis. It highlights the challenges LLMs face in identifying correct values from a large set of possibilities, particularly when dealing with categorical values that are not inherently recognized by the model. The document details different strategies tested to improve the accuracy and efficiency of query analysis, including context stuffing, pre-LLM filtering, and post-LLM selection, with a focus on methods like embedding similarity and n-gram similarity for filtering and selecting valid names. The results indicate that post-LLM selection using embedding similarity offers the best performance in terms of accuracy, speed, and cost. The study emphasizes the need for further benchmarking with even larger datasets typical in enterprise systems, which often deal with millions of possible values.

Interest in long-context language models (LLMs) is growing as context windows expand to accommodate up to 1 million tokens, prompting the development of new benchmarks to evaluate their capabilities. One such benchmark, the Multi-Needle + Reasoning benchmark, tests the ability of LLMs to retrieve and reason over multiple facts within a large context. The results show that performance degrades as the number of facts (needles) increases, and the challenge intensifies when reasoning is required. Notably, models like GPT-4 tend to retrieve facts placed towards the end of the context while ignoring those at the beginning, a pattern seen in both single and multiple needle setups. As context length grows, retrieval and reasoning performance both decline, highlighting potential limitations of LLMs when used in retrieval-augmented generation (RAG) applications. Understanding these limitations is crucial for effectively utilizing long-context LLMs, as they may not guarantee retrieval of multiple facts, especially as the context size increases, and specific prompting strategies might be needed to enhance performance.

Dosu, an AI teammate designed to help develop, maintain, and support software, was created by Devin Stein, CEO of Dosu, to alleviate the burden on open source software maintainers and developers who often spend a significant portion of their time on non-coding tasks. Dosu uses Evaluation Driven Development (EDD) to ensure reliability and performance, continuously iterating on its logic and models to handle various scenarios effectively. As Dosu's usage grew, monitoring its activity and identifying failure modes became challenging, leading to the integration of LangSmith, a tool that provides enhanced monitoring capabilities through its SDK, allowing for detailed tracing and evaluation of Dosu's performance. LangSmith's features, such as customizable metadata tracking and advanced search functionalities, have enabled Dosu to identify and address unforeseen failure modes, thus improving its service. The collaboration between Dosu and LangChain, facilitated by LangSmith, creates a synergistic cycle where improvements to Dosu reduce maintenance burdens, allowing for more focus on feature development, ultimately enhancing both platforms.

A new open-source extraction service has been launched, designed to streamline the process of extracting structured data from unstructured sources using Large Language Models (LLMs). This service builds on the capabilities of the LangChain library and includes a starter repository to assist users in developing their own extraction applications, featuring a web application that can be extended for non-technical users. It addresses the challenges faced by enterprises in extracting valuable insights from varied document types by replacing traditional rule-based and complex ML-based models with LLM-driven solutions, which are easier to maintain and scale. The service employs FastAPI and Postgresql to offer a REST API, enabling the creation of "extractors" that define the schema, prompt, and reference examples for the LLM. It supports MIME-type based parsing of PDFs and HTML files and can be extended to other formats. The service is designed to extract multiple entities from text, allowing for flexibility and scalability in information extraction. Additionally, it offers a practical example of use, demonstrating how to register and invoke an extractor to retrieve structured data from text inputs.