December 2023 Summaries

In 2023, the surge in interest in Generative AI, sparked by ChatGPT, led companies to explore strategies for incorporating this technology into their products. LangSmith, a cloud platform from LangChain, has been instrumental in helping teams transition from prototype to production by providing features like tracing, regression testing, and evaluation. The platform's usage data reveals insights into the construction and testing of LLM applications, with retrieval emerging as a key method for combining data and 42% of complex queries involving retrieval. The introduction of LangChain Expression Language (LCEL) has facilitated the creation of complex, customized chains, reflecting a trend towards experimentation and customization in the GenAI landscape. OpenAI is the leading LLM provider, with AzureOpenAI and other proprietary and open-source models also gaining traction. Vectorstores play a crucial role in LLM applications, with local vectorstores being most popular, and Pinecone and Weaviate leading among hosted options. Testing and evaluation are major challenges, with LangSmith enabling users to formulate various metrics, predominantly using LLMs for output evaluation, although custom evaluators are common due to the specificity required by different applications. As the year concludes, LangSmith is establishing itself as a key tool for teams aiming to move from prototype to production in the evolving field of LLM app development.

As 2023 concludes, Rubric's Graphite team aimed to celebrate GitHub users' contributions by creating "Year in Code," an AI-generated, personalized video experience. Unlike previous projects, this initiative utilized LangChain and OpenAI's GPT-4-turbo to transform GitHub user data into unique narratives, moving beyond static images and templated formats. The project employed cutting-edge technologies such as Supabase for authentication and data storage, Remotion for video creation, and AWS Lambda for video rendering. By focusing on a structured video_manifest and leveraging pre-built scene templates, the team balanced AI creativity with quality control, resulting in engaging and shareable 60-second videos. This open-source project, which emphasizes user privacy by excluding code access, showcases the potential of AI and modern tech stacks in generating immersive and personalized digital experiences.

The release of four new test environments aims to benchmark large language models' (LLMs) ability to effectively use tools for task completion, focusing on skills essential for agentic workflows such as planning, function calling, and overriding pre-trained biases. The tests, including tasks like the Typewriter (single and 26 tools), Relational Data, and Multiverse Math, reveal that while models like GPT-4 excel in some areas, they struggle with others, particularly when tasks require multi-step reasoning or deviate from pre-training patterns. Findings indicate that even high-performing models can fail on seemingly simple tasks when faced with complex tool usage scenarios, and reliability issues such as frequent server errors from model providers pose challenges for consistent performance. The research underscores the need for open-source function calling models with improved multi-step task handling and highlights the importance of validating LLMs against specific task requirements before deployment.

Retrieval augmented generation (RAG) is crucial in the development of large language model (LLM) applications, enabling interactive chats and question-answering systems with documents. The text outlines three strategies for integrating semi-structured data like tables into LLM contexts: directly inputting documents into long-context LLMs, targeted extraction of tables, and document chunking. While long-context LLMs offer simplicity, they struggle with large datasets and information placement within inputs. Targeted table extraction, despite its complexity, potentially offers the highest performance for complex tables but requires specialized tools. Document chunking, although straightforward, faces challenges in maintaining table integrity unless chunks align with page boundaries. The use of ensemble retrievers can enhance the retrieval of table-derived information by prioritizing their chunks over text body chunks, thereby improving the performance of LLM-driven applications in handling structured data.

LangChain, initially a side project and Python package, has evolved significantly over the past year, prompting a re-architecture into three distinct packages to enhance developer experience, maintainability, and scalability while remaining backwards compatible. The new structure includes langchain-core, which houses core abstractions and the LangChain Expression Language to facilitate composability, langchain-community for third-party integrations, and langchain for high-level cognitive architecture components. This reorganization aims to stabilize the ecosystem, simplify versioning, and improve integration management. LangChain also supports an expansive ecosystem with tools like LangServe and LangSmith, which aid in deployment and debugging, respectively. This strategic shift encourages further exploration and optimization of use cases in the GenAI space, allowing integration partners to own their frameworks and applications more fully while supporting an ever-growing user base.

Retrieval augmented generation (RAG) in large language model (LLM) app development has expanded to include visual content from slide decks, enabled by multi-modal LLMs like GPT-4V. This advancement allows for interactive chat and Q&A by retrieving and synthesizing information from visual data. Two main approaches to multi-modal RAG in slides are multi-modal embeddings, which involve embedding slide images to retrieve relevant content, and multi-vector retrievers, which summarize images before retrieval. While multi-modal embeddings offer simplicity, they face challenges in distinguishing visually similar slides, whereas image summarization, though more complex and costly, enhances retrieval accuracy. A public benchmark evaluation using a Datadog presentation demonstrated that multi-modal methods significantly outperform text-only RAG, achieving accuracy scores of 60% and 90% compared to 20% for text-only approaches. The study also highlighted the effectiveness of GPT-4V in extracting structured data from images and emphasized the importance of accurate image retrieval for successful question answering. To support further exploration and deployment, a template leveraging Chroma and OpenCLIP multi-modal embeddings has been released.

The LangChain team has introduced a new extraction dataset designed to evaluate the ability of Large Language Models (LLMs) to extract structured information from chat logs, which aims to address common challenges in LLM application development such as classifying unstructured text and reasoning over multi-task scenarios. The dataset's schema is crafted to extract structured insights from chatbot interactions and has been tested using various LLMs, including closed-source models like GPT-4 and Claude-2, as well as open-source models like Llama 2 and models from Nous Research. While GPT-4 generally outperforms others in structured output, open-source models like Llama 2 show varying degrees of success based on model size and fine-tuning. The study also examines the effectiveness of different prompting strategies and structured decoding techniques, revealing that while structured decoding guarantees schema compliance, it does not necessarily improve the quality of the extracted values. The findings highlight the challenges in achieving consistent and accurate structured information extraction from chat data, suggesting a need for further refinement in model training and prompting strategies.

InstaMortgage, in collaboration with Aptford and using LangChain's technology, has revolutionized the mortgage industry by implementing an AI platform, InstaAI, which consists of three core modules: Mortgage AI, Content AI, and Policy AI. These modules streamline the loan application process by offering accurate guidance, creating informative content, and clarifying company policies, respectively. For loan officers like Maya, InstaAI has transformed their roles from managing tedious tasks to building substantial client relationships, significantly enhancing efficiency and client satisfaction. The AI platform has reduced mortgage processing times, decreased error rates, and increased client satisfaction by an average of 67% in speed to resolution, with employees now able to engage in more rewarding tasks. InstaMortgage's CEO, Shashank Shekhar, highlights the company's commitment to expanding its AI capabilities to further enrich both employee and client experiences.