November 2023 Summaries

Large language models (LLMs) are conceptualized as the core process of a novel operating system, with a context window that acts like RAM, enabling information retrieval from various sources for output generation. This process, known as retrieval augmented generation (RAG), is pivotal in LLM application development, offering a more straightforward alternative to complex fine-tuning for tasks requiring factual recall. The landscape of RAG methods is rapidly evolving, leading to some user confusion, prompting efforts to categorize and guide their use. Key RAG themes include query transformations to enhance retrieval robustness, dynamic query routing across diverse data stores, query construction using text-to-SQL or text-to-Cypher for structured data, and indexing strategies like optimizing chunk size and document embedding. Post-processing of retrieved documents is crucial due to the limited context window, with methods like re-ranking and classification to improve document diversity and relevance. Future plans involve leveraging open-source models for specific RAG tasks and developing benchmarks using public datasets to evaluate these approaches.

OpenGPTs, an open-source implementation of OpenAI GPTs and the Assistant API, facilitates the creation of conversational agents with a focus on memory, particularly long-term memory, which is often underexplored in current implementations. The blog discusses various types of memory, including conversational and semantic memory, highlighting the limitations of stateless large language models (LLMs) that require users to manage state externally. It also explores the concept of generative agents, which utilize recency, relevancy, and reflection to enhance memory capabilities. A specific application is demonstrated through a Dungeons and Dragons chatbot, which employs a custom memory system to track character and quest information, utilizing a state machine architecture to transition between gathering character information and leading quests. The blog emphasizes the importance of application-specific memory in enhancing reliability and performance and introduces tools like LangChain and LangSmith to assist in building such systems, acknowledging the complexity involved in developing long-term memory solutions tailored to specific applications.

OpenAI's recent developer day unveiled the Assistants API and GPTs, both aimed at advancing applications towards a specific "agent" cognitive architecture. These technologies embody OpenAI's ambition to create transformative, agent-like systems. While LangChain sees potential in such architectures, it emphasizes the importance of customizable, open systems, as evidenced by its OpenGPTs project, which offers an open, configurable alternative to OpenAI's closed systems. The Assistants API is developer-focused, allowing users to control tools client-side, while GPTs offer a no-code approach to creating custom "GPTs." Despite the promise of these innovations, OpenAI is betting on overcoming the current reliability issues associated with agent architectures. This bet is contrasted with other applications like GPT-Researcher and Sweep.dev, which use complex chains or state machines to maintain control and performance. The discourse around open vs. closed cognitive architectures is gaining traction, with LangChain advocating for open systems to ensure flexibility and control, which are crucial for developing robust, scalable applications.

Adyen, a financial technology platform, is enhancing the efficiency and satisfaction of its support teams by implementing innovative solutions using large language models (LLMs). To address the challenge of ticket response times, Adyen established a team of Data Scientists and Machine Learning Engineers at their new Tech Hub in Madrid to develop a smart ticket routing system and a support agent copilot. These tools aim to direct tickets to the appropriate support person swiftly and aid agents in providing faster, more accurate responses. By leveraging LangChain, Adyen built a flexible framework that allows for easy customization and experimentation with various LLMs. This approach has improved ticket routing accuracy and reduced response times by dynamically analyzing ticket themes and sentiments and utilizing a vector database for efficient document retrieval. The integration of these technologies into a microservice architecture hosted on Kubernetes has resulted in more efficient support operations and increased agent satisfaction.

LangSmith addresses developers' challenges in app testing and evaluation by providing a comprehensive platform for evaluating large language model (LLM) architectures. It offers features like dataset sharing, benchmarking through the langchain-benchmarks package, and detailed evaluation results, including traces for each tested chain to facilitate community-driven evaluations. The initial benchmark focuses on a Q&A dataset over the LangChain Python documentation, evaluating various LLM architectures like OpenAI's GPT models and Anthropic's Claude. LangSmith helps users compare different models and architectures by providing metrics like cosine distance and accuracy scores, enabling developers to choose the best solutions for their applications. It also highlights the importance of latency and performance trade-offs, with the LangChain community contributing to the rapid evolution of LLM tooling and model quality. Additionally, LangSmith encourages experimentation through public datasets and evaluations, aiding developers in staying updated with advancements in the field.

Calix, a sophomore at USC Iovine & Young Academy, developed Dream, an AI-driven no-code tool designed to help both technical and non-technical users create fully functional web apps and components using natural language. During a 10-week hacker residency at LangChain, Calix focused on enhancing Dream's performance, extensibility, and generation quality by addressing challenges such as user experience and custom integrations. He experimented with different user interaction methods, ultimately settling on a survey-based approach to guide users in articulating their project requirements, and incorporated Claude 2 to improve the handling of custom API integrations. A significant shift involved migrating from HTML/JS to React for better efficiency and versatility, supported by LangChain's guidance. Calix also refined the prompting strategy inspired by open-source projects, enhancing Dream's design flexibility. Despite the challenges of creating an intuitive interface for code generation, Calix remains committed to advancing Dream as a platform to empower non-technical users in software development.

Tuna is a no-code tool designed to enable the rapid creation of high-quality fine-tuning datasets for large language models (LLMs) like GPT-3.5-turbo and LLaMa-2, facilitating the process of training models for specific applications or domains. Available via a web interface and a faster Python script, Tuna allows users to generate prompt-completion pairs by inputting a CSV file of text data, which is processed through OpenAI's API to minimize hallucination. Fine-tuning LLMs is valuable for adapting them to particular tasks, such as legal writing or conversational formats, by specializing their responses and enhancing their performance on smaller, self-hosted models. While fine-tuning can be resource-intensive due to the need for high-quality datasets, Tuna lowers these barriers by automating the generation of synthetic datasets. This tool supports various configurations for dataset creation, including SimpleQA, MultiChunk for retrieval-augmented generation (RAG), and custom prompts, providing flexibility in tailoring data for specific fine-tuning purposes. Fine-tuning can improve response reliability and formatting, though its efficacy in embedding new information remains debated, with RAG often providing a more practical solution.

OpenAI's demo day showcased various Retrieval-Augmented Generation (RAG) experiments, highlighting that different retrieval techniques suit different problems. Their study demonstrated the efficacy of methods like distance-based vector database retrieval, which uses cosine similarity for document matching, and query transformations such as LangChain's Multi-query retriever and HyDE, which improve retrieval by generating multiple perspectives or hypothetical documents. Routing questions appropriately across multiple datastores, including SQL databases, is crucial, and LangChain supports such routing with LLMs. Building the index with optimal chunk sizes and employing post-processing techniques like re-ranking and classification can enhance retrieval performance. OpenAI's experiments underscore the importance of evaluation to ensure effective RAG approaches, with tools like LangSmith available to support this process.

A new LangChain template, developed in collaboration with the GPT Researcher team, offers a research assistant tool that diverges from the typical chat-based user experience (UX) of many language model applications. This template focuses on creating long-form research reports by generating sub-questions, retrieving and summarizing relevant documents, and combining these into a final report, all of which is facilitated within the LangChain ecosystem. Unlike chat applications, which are bound by latency expectations and often require a human-in-the-loop for accuracy, this template allows for more complex and autonomous operations, providing users with high-quality outputs that can be inspected and modified. The integration with tools like LangSmith enhances observability by allowing users to track the process comprehensively. The template uses OpenAI and Tavily, a search engine optimized for AI workloads, but is customizable to work with various data sources. This shift towards non-chat, longer-running applications aims to meet growing demands for sophisticated AI solutions that prioritize quality over speed, embodying a trend towards more autonomous agent frameworks in AI development.

LangChain and Microsoft are collaborating to integrate their technologies, enhancing the capabilities of AI-driven applications for joint customers. This partnership aims to leverage LangChain's expertise in context-aware reasoning applications and Microsoft's robust Azure ecosystem, offering deeper product integrations and enterprise-grade assurances. LangChain has joined the Microsoft for Startups Pegasus Program, which will enable easier procurement of their LangSmith SaaS product via the Azure Marketplace, providing benefits such as improved developer productivity and enhanced data security within customers' Azure Virtual Private Clouds. The collaboration focuses on streamlining application development processes and expanding the integration of tools like Azure OpenAI Service, Azure AI Search, and Microsoft Fabric, making it easier for enterprises to adopt and scale AI technologies effectively.

Morningstar, an investment research firm established in 1984, has expanded its offerings by integrating Generative AI to improve data accessibility and usability for financial professionals. By developing the chatbot Mo, users can now interact with Morningstar's extensive research database using natural language for rapid insights. The company further enables asset managers and wealth advisors to create customized AI tools through the Morningstar Intelligence Engine, allowing these professionals to personalize and secure their data interactions. The development leveraged LangChain due to its robust framework for large language model-powered applications, facilitating a quick transition from concept to production. This integration of AI technology, completed in less than 60 days by a small team, aligns with Morningstar's long-standing commitment to delivering innovative tools and insights that enhance investment decision-making.

The blog post explores the challenges and strategies involved in converting natural language queries into structured query syntax for different types of data, including structured, semi-structured, and unstructured data. It highlights the concept of query construction, which translates user queries into appropriate database query languages, such as SQL for structured data and Cypher for graph databases. The post emphasizes the role of language models (LLMs) in improving retrieval-augmented generation by constructing precise queries that leverage both the structure of the data and semantic understanding. It discusses the challenges of hallucinations and user errors in text-to-SQL conversions and suggests methods to overcome them, such as providing accurate database descriptions and using few-shot learning. Additionally, it introduces the integration of vector support with relational databases, enabling hybrid retrieval approaches that combine semantic and structured searches. The article provides insights into the use of knowledge graphs for modeling complex relationships and mentions the potential of using advanced LLMs like GPT-4 for generating valid Cypher queries. Overall, the post underscores the importance of developing effective natural-language-to-structured query systems to maximize the capabilities of LLMs across various data sources.

Parallel function calling, introduced by OpenAI in 2023, enhances the process of extracting structured data from unstructured documents using language models by allowing multiple function calls to be made simultaneously, thereby simplifying the extraction process. Previously, developers had to implement complex workarounds to extract multiple pieces of information at once, often involving intricate typing logic and parsing. With parallel function calling, the need for such hacks is eliminated, as the model can directly handle multiple data types, such as Person and Location, without requiring additional layers of logic. This not only improves the developer experience by reducing the complexity of input and output logic but also decreases the likelihood of errors in generating or parsing JSON outputs. The approach significantly streamlines the task of entity extraction, making it more efficient and less error-prone while also highlighting the broader applications of function calling in structuring language model outputs.

SPADE (System for Prompt Analysis and Delta-based Evaluation) is a tool developed by researchers at UC Berkeley in collaboration with LangChain to enhance the evaluation of Large Language Model (LLM) chains by leveraging prompt refinement history. The tool suggests Python-based evaluation functions that can assess the quality and reliability of LLM outputs by identifying changes in prompt versions and categorizing them based on a developed taxonomy. SPADE aims to address the challenges of prompt engineering and monitoring in LLM deployments by offering automated evaluation functions that can verify the adherence to constraints and guardrails encoded in prompt refinements. The prototype suggests evaluation functions by analyzing the differences between prompt versions, which are useful in ensuring LLM outputs meet specified criteria, such as excluding certain items in response to a specific context. Despite being in a preliminary stage, SPADE offers potential improvements in LLM deployment reliability and invites feedback and collaboration from developers interested in this research area.

The blog post explores advanced retrieval strategies for retrieval-augmented generation (RAG) applications, emphasizing the enhancement of vector similarity searches by incorporating contextual understanding. It highlights the use of Neo4j, a graph database, to manage document hierarchies and introduces a LangChain template that supports multiple RAG strategies. These strategies include splitting large documents into smaller vectors for better indexing, generating hypothetical questions and summaries for indexing, and ensuring the retrieval of parent documents to maintain context. The post also provides a guide to setting up a Neo4j environment, using LangChain templates, and deploying the neo4j-advanced-rag template with LangServe to facilitate the comparison of different RAG strategies. The strategies aim to improve the accuracy and relevance of information retrieved by large language models (LLMs) by balancing precise embeddings with context retention.

The choice of embedding models significantly affects the quality of chatbots using Retrieval-Augmented Generation (RAG), as demonstrated by the Voyage AI team's analysis of Chat LangChain, a chatbot that answers questions about LangChain documentation. RAG combines a retrieval system with a generative model, and the effectiveness of the system depends heavily on the quality of embeddings, which transform queries and documents into vectors for semantic search. The study compares three embedding models: OpenAI's text-embedding-ada-002, Voyage's generalist model voyage-01, and a fine-tuned version voyage-langchain-01, specifically tailored for LangChain documents. The evaluation, using metrics such as retrieval quality and end-to-end response quality, shows that the fine-tuned voyage-langchain-01 outperforms the others, highlighting the importance of specialized embeddings in improving both retrieval and response accuracy. The post further demonstrates how better retrieval with Voyage's embeddings leads to more accurate responses, especially in domain-specific queries, and announces the integration of Voyage into the LangChain Python package for broader accessibility.