October 2023 Summaries

LangChain Templates is a newly introduced collection of easily deployable reference architectures designed to expedite the production of context-aware reasoning applications. These templates, developed in collaboration with partners, are aimed at helping developers create, share, maintain, and customize chains and agents efficiently. They are formatted for seamless deployment with LangServe, facilitating the creation of production-ready APIs and offering a free playground. The initiative addresses developers' needs to understand, modify, and deploy application internals without causing bloat in the main LangChain package, by allowing direct access to source code through downloads instead of installations. LangChain Templates also supports various models, including open-source ones, and caters to use-case-specific chains, enhancing flexibility and innovation in application development. The platform integrates with LangSmith for logging and debugging and plans to offer a hosted version of LangServe for streamlined deployments.

LangSmith, a developer platform designed to facilitate the transition of large language model (LLM) applications from prototype to production, introduces Data Annotation Queues to enhance application performance and reliability. This feature allows users to review and annotate logs, facilitating the creation of datasets essential for refining LLMs. It addresses the challenge of needing application-specific data by enabling easy data exploration and curation. Additionally, LangSmith highlights langfree, an open-source package by Hamel Husain, which complements Data Annotation Queues by offering customizable annotation and dataset curation workflow capabilities. These tools aim to provide teams with the means to build high-quality, application-specific datasets, crucial for improving LLM applications.

LangChain offers advanced retrieval methods to enhance the retrieval-augmented generation (RAG) process by addressing challenges like irrelevant content in document chunks, poorly worded user queries, and the need for structured query generation. This approach leverages large language models (LLMs) to perform query transformations, enabling the rewriting of user queries and generating search terms that improve retrieval accuracy. Strategies include multi-representation indexing, query transformation to enhance the user's original question, query construction for specific query syntax, and multi-query retrieval that generates multiple sub-queries for complex questions. By using LLMs, these methods introduce novel possibilities in query transformation, relying heavily on the prompts used to guide the LLMs, and opening new avenues for prompt engineering to optimize retrieval results.

LangChain celebrated its first anniversary, marking a year of significant growth and collaboration in the field of Generative AI. Over 2,000 developers contributed to its Python and JavaScript libraries, with more than 20 million developers downloading the open-source package to create over 30,000 applications. The platform has integrated with a wide array of tools, including 150 document loaders, 70 language models, and numerous vector stores, embedding models, and chat models. The community-driven Prompt Hub has seen the addition of hundreds of prompts, enhancing the ecosystem. Contributions from key developers and educators have been acknowledged, alongside influential papers that have propelled the understanding of Generative AI. The LangChain team expressed gratitude to its global community for their support and innovation, highlighting the collaborative efforts that have driven LangChain's success in its inaugural year.

Norwegian Cruise Line's Andres Torres and Dylan Brock explore the challenges and nuances of designing UI/UX for AI applications, particularly focusing on bridging the gap between traditional user interfaces and the immersive experiences that Generative AI (GenAI) can offer. They highlight the importance of creating a seamless user journey from the first digital interaction to the end of a vacation experience, emphasizing that current GenAI products often fail to inspire users adequately. The authors discuss the need for user-friendly shortcuts, anthropomorphism in AI interfaces, and the critical role of user feedback in refining AI applications. They argue that while chat-based UIs are familiar to users, they often lack the personal touch of human interaction and can overwhelm users with their complexity, creating barriers to entry. The post suggests practical strategies for improving user engagement with GenAI, such as embedding shortcuts, considering anthropomorphic design elements, and integrating direct and indirect feedback systems to enhance the accessibility and usability of AI-driven platforms.

Founded in 2019, Robocorp aimed to fulfill the unmet potential of automation in software development by introducing a Python-based platform that enhanced automation efficiency. To further elevate their offerings, they developed ReMark, an AI-powered developer assistant designed to generate code, address specific queries, and streamline the automation process for developers and non-developers alike. This tool, described as akin to a seasoned developer familiar with Robocorp's tools, leverages LangChain's capabilities to integrate data sources and utilize extensive automation examples and documentation. This has significantly expedited bot development, reportedly making the process four times faster, and has reduced support hours by enabling customers to find solutions independently. The introduction of ReMark has not only accelerated Robocorp's business growth but also allowed the company to maintain high levels of customer support while focusing on innovative product development.

Seamless question-answering across diverse data types, including images, text, and tables, has been a major objective in the field of Retrieval-Augmented Generation (RAG), with recent advancements focusing on multi-vector retrieval techniques. Three new cookbooks have been released to demonstrate the application of these techniques on documents with mixed content types, highlighting the potential of multimodal models like GPT4-V and LLaVA for enhancing RAG capabilities on images. RAG allows Large Language Models (LLMs) to enhance their factual recall by integrating reasoning capabilities with external data sources, beneficial for enterprise data handling. Techniques to improve RAG involve various strategies such as metadata filtering and multi-stage retrieval processes. The multi-vector retriever, introduced earlier, enables the decoupling of documents from references, allowing for a more efficient answer synthesis process without losing context, and can be applied to both semi-structured data and multiple modalities. The document also discusses how to partition documents by types using tools like Unstructured, which supports RAG by extracting tables, images, and text from various file formats. This approach allows for the generation of summaries optimized for retrieval, passing the full document context to LLMs when necessary. Moreover, the cookbooks propose methods for using multimodal embeddings and LLMs to process image data within RAG, offering privacy-conscious solutions that can be executed locally with open-source components.

LangServe has been launched to facilitate the deployment of chains and agents by creating a FastAPI server with streaming and batch endpoints, and now includes a playground and configurability for enhanced experimentation and collaboration. The playground feature offers a simple UI for users to change configurable parameters, test different inputs, and view streaming outputs, thus enabling non-technical users to interact with the applications. Configurability is augmented by allowing components to be configurable via LangChain Expression Language, supporting both internal and external applications, and promoting experimentation with different models and retrievers. This approach aligns with emerging common architectures for LLM applications, focusing on retrieval-augmented generation, and aims to simplify initial engineering efforts while encouraging configuration through a graphical user interface for improved collaboration.

Tomaz Bratanic from the Neo4j team discusses the advancements in extracting structured information from unstructured text using Large Language Models (LLMs), highlighting how the process has become more accessible thanks to these models. The article explores constructing a knowledge graph from a sample Wikipedia page using OpenAI functions and LangChain, emphasizing best practices and limitations of current LLMs. It outlines an information extraction pipeline that includes steps like coreference resolution, named entity recognition, and entity disambiguation, and demonstrates how a LangChain-based setup can connect to a Neo4j database to build the knowledge graph. The text also underscores the importance of defining graph schemas and the entity disambiguation step for accuracy, while showcasing how to query the graph using Cypher statements in a Retrieval-Augmented Generation (RAG) application. The author concludes by inviting readers to learn more about AI applications with graphs at the upcoming NODES conference organized by Neo4j.

Prompt engineering allows for the manipulation of Large Language Model (LLM) behavior without altering the model itself, leading to a rise in diverse prompt types for various applications. The LangChain Hub was introduced to facilitate prompt management, offering a platform for discovering, sharing, and refining prompts. Popular prompt categories include reasoning, writing, content generation, SQL interfacing, brainstorming, and extraction, each serving different functions such as enhancing reasoning skills, improving writing clarity, generating diverse content, and extracting structured data. Retrieval augmented generation (RAG) combines LLM reasoning capabilities with external data for factual recall, while instruction-tuned LLMs and LLM graders provide tailored solutions for specific tasks. The creation of synthetic data for fine-tuning and prompt optimization further showcases the potential of LLMs in generating innovative outputs. Additionally, LLMs are instrumental in code understanding and generation, leveraging platforms like GitHub co-pilot. Summarization remains a key application, with advanced techniques enabling the condensation of extensive content into concise, high-quality overviews. Users can experiment with these prompts in an interactive playground, offering practical exposure to a variety of LLMs.

You.com has announced the release of YouRetriever, which provides access to their You.com Search API, designed specifically for Retrieval Augmented Generation (RAG) applications. This API has been evaluated against various datasets to benchmark its performance, particularly in the RAG-QA setting, where it is compared with the Google Search API. The Hotpot QA dataset is used to test how well the API retrieves relevant information, even in the presence of distractor text. The You.com API is noted for its ability to return more extensive and numerous text snippets, which is advantageous for language models with longer context windows. A comparison shows that the You.com Search API significantly outperforms Google's API on a test dataset, with plans for a larger study to be released soon. Early access to the API is available for interested partners who can provide details about their use case and expected usage.

In a blog post by Martin Zirulnik, the focus is on enhancing context-aware language model applications through a novel approach to text chunking using HTML structure. The post introduces the HTML Header Text Splitter, a tool that respects document hierarchy by splitting text at the element level, preserving contextual information often lost in traditional web-scraped data. This method is contrasted with conventional arbitrary chunking, revealing its limitations in maintaining context precision and recall. The blog demonstrates how structured chunking, combined with LangChain's self-querying retriever, can significantly improve results in Retrieval Augmented Generation (RAG) applications by leveraging document structure for more precise and contextually relevant information retrieval. It highlights the importance of semantic memory utilization in enhancing generative AI's ability to maintain authority and preserve context, particularly in critical fields such as education, medicine, and law.

LangChain's LangSmith platform introduces a feature called Test Run Comparisons to improve the evaluation of large language model (LLM) applications, addressing the challenge of quantitatively assessing changes to prompts, chains, or agents. This feature allows users to manually inspect and compare multiple test runs within a dataset, providing a user-friendly interface to view inputs, reference outputs, actual outputs, and evaluation metrics. Users can apply filters to focus on significant differences between test runs, aiding in the discovery of changes and enhancing understanding of the LLM's performance on specific tasks. By facilitating side-by-side comparisons and enabling deeper exploration of datapoints, LangSmith aims to build infrastructure that supports manual data inspection, aligning with the practices of successful AI researchers and engineers. Currently in private beta, LangSmith invites feedback as it plans to expand access and introduce more features.

Ryan Brandt, CTO and co-founder of ChatOpenSource, discusses using LangSmith, a platform by LangChain, to bring Large Language Model (LLM) applications into production effectively. He highlights the increasing capability of open-source models like Mistral 7b and Llama2, emphasizing the future potential of easily swapping models in applications. Brandt outlines a process for fine-tuning and evaluating models, such as Llama2-7b and Llama2-13b, using LangSmith to automate evaluations and compare performance across different models using datasets. The LangSmith platform streamlines dataset evaluation with a user-friendly UI and API, enabling developers to assess model performance efficiently. Brandt's findings reveal a relationship between model parameters, training data volume, and performance, noting that llama2 models can perform comparably to GPT-3.5-turbo-base under certain conditions. This process underscores the potential of open-source models to compete with established models and the importance of efficient model evaluation tools like LangSmith.

Rubric Labs developed an AI-driven email assistant, Cal.ai, designed to streamline calendar management through natural language processing and integration with Cal.com's API. Utilizing an OpenAI functions agent through LangChain, the system processes user requests sent via email, converting them into actionable scheduling tasks. This is achieved by employing a combination of fine-tuned GPT models and dynamic structured tools, enabling the execution of CRUD operations on user calendars. The architecture involves parsing incoming emails, verifying them, and using a chat model to interact with various tools to fulfill user requests. The system emphasizes consistent and reliable interaction by using a low-temperature setting on the GPT-4 model to minimize output variability. Once the agent processes a request, it sends the user a response via email, facilitated by services like Sendgrid. This approach showcases the efficacy of combining language models with structured tools to handle natural language input effectively, positioning AI agents as powerful solutions for managing structured data tasks.

The blog post discusses the development of a web application that utilizes local machine learning models to perform Retrieval-Augmented Generation (RAG), enabling users to interact with documents through chat. The author explores building this app using JavaScript to capitalize on its widespread use among web developers, aiming to leverage local models for cost efficiency, privacy, and potentially faster processing due to reduced server requests. The process involves splitting documents into semantic chunks, creating vector representations with embeddings, and storing them in a vector store to facilitate natural language queries. Despite challenges with running large language models (LLMs) in the browser, the author successfully integrates the Mistral 7B model using Ollama, demonstrating that local models can be effectively used in web apps with the right configuration. The post highlights the rapid advancements in open-source machine learning models and the potential for future web applications to incorporate local LLMs through new browser APIs, while also providing links to resources and further reading.

LangServe is a newly launched tool designed to streamline the deployment of LangChain applications, enabling developers to easily transition from prototype to a production-ready API. Built as a Python package, LangServe leverages the LangChain Expression Language (LCEL) and the Runnable protocol to support efficient deployment of complex chains, with features like first-class streaming, async support, optimized parallel execution, and support for retries and fallbacks. It also introduces input and output schemas for validation and offers hosting solutions on platforms like GCP and Replit. The tool has been tested with applications like ChatLangChain and WebLangChain and aims to provide a seamless and scalable path for launching AI-driven applications. Future updates will include new functionalities, such as a playground for experimenting with prompts and retrievers, and support for saving multiple configurations of the same chain.

Fine-tuned ChatGPT has demonstrated superior performance over GPT-4 for news article summarization by using synthetic data and advanced evaluation methods like the ScoreStringEvalChain and PairwiseStringEvalChain. While GPT-4 is highly regarded for its language capabilities, challenges such as high costs, latency, and deployment difficulties have led developers to explore alternative models like ChatGPT. Fine-tuning involves adjusting model weights to improve task-specific performance, and in this study, the chain of density prompting was used to iteratively enhance summaries, making them more information-dense and favored by humans. The fine-tuned ChatGPT surpassed GPT-4's zero-shot performance while being significantly faster and cheaper, achieving a 96% win rate in pairwise evaluations. The study validates using synthetic data and automated evaluation systems to refine language models, offering a cost-effective and efficient solution for real-world applications, particularly through tools like LangChain and LangSmith, which facilitate the creation and evaluation of complex AI workflows.

The blog post discusses the process of building an open-source web research assistant powered by Tavily, focusing on engineering decisions involved in creating applications that connect Large Language Models (LLMs) to external knowledge sources using Retrieval Augmented Generation (RAG). The article details the challenges and decisions in the retrieval step, such as whether to always perform lookups, handle follow-up questions, and manage multiple search terms or lookup steps. It also outlines the use of the Tavily Search API for retrieving snippets and highlights the importance of generating search queries to improve response accuracy. For the augmented generation step, the article explains the choice of using GPT-3.5-Turbo for its cost-effectiveness and speed, as well as the decisions around prompt design and providing sourced responses to enhance reliability and allow deeper exploration of information. The blog post aims to provide insights into the tradeoffs of engineering decisions in RAG applications and offers a starting point for developers with a shared code repository.

The post explores the integration of knowledge graphs into Retrieval-Augmented Generation (RAG) applications, particularly for enhancing chatbots' ability to handle both structured and unstructured data. It highlights the use of Neo4j to store and manage information regarding microservices architecture and tasks, utilizing nodes and relationships to encapsulate entities and their interactions. The post demonstrates the implementation of a vector similarity search and a Cypher-based query system to retrieve relevant data efficiently, showcasing the strengths and limitations of each method. Furthermore, it introduces the use of LangChain to facilitate a seamless interaction between vector and graph queries, thereby improving data retrieval capabilities in RAG applications. The post emphasizes the advantage of using knowledge graphs to avoid the complexity of managing multiple databases while enabling sophisticated data-driven AI applications.

Kay and Cybersyn have collaborated to simplify financial data processing from SEC Filings for developers using Retrieval Augmented Generation (RAG) in generative and conversational agents. By addressing challenges such as the rapid evolution of embedding infrastructure, complex financial document formats, and the need for up-to-date data, they offer a system that provides enriched, pre-embedded datasets for efficient retrieval. The SEC Retriever on LangChain leverages Kay's data APIs to provide context from SEC Filings, while Cybersyn supplies analytics-ready economic data via Snowflake Marketplace. The infrastructure includes high-quality data collection, dynamic embedding generation, and optimized retrieval processes, making it easier for developers to access and utilize financial document data in real-time. This initiative enables various users, including analysts and investors, to quickly parse and analyze financial information, enhancing decision-making processes.

Francisco, a founder at Pampa Labs, discusses the challenges and solutions for handling personally identifiable information (PII) when using large language models (LLMs) like those provided by OpenAI and other companies. With the rising importance of regulations such as GDPR, it's crucial to anonymize PII to prevent potential data leaks. Tools like Microsoft Presidio and OpaquePrompts are highlighted for their efficiency in masking PII within the LangChain ecosystem. Presidio uses a combination of rule-based logic and machine learning to identify and anonymize PII, while OpaquePrompts employs a single ML model and confidential computing to protect data privacy. Additionally, strategies for managing PII when logging app conversations with LangSmith are discussed, including hiding or masking inputs and outputs. The post emphasizes the importance of staying informed about providers' privacy policies and integrating innovative methods for maintaining data privacy in LLM applications.

A year ago, OpenAI's GPT-3 was the predominant large language model (LLM), but now there is a variety of models available, including many open-source options that have significantly improved in performance. In response to growing interest in these models, Fireworks AI has partnered with LangSmith to integrate their models into the LangSmith playground, allowing users to access them for free without needing an API key, provided they are signed into the platform. Fireworks AI offers a platform for developers to run, fine-tune, and share LLMs, optimizing for latency, throughput, and cost efficiency, with claims of up to 120 times cost reduction. This integration makes it easier for users to experiment with and optimize prompts using high-performing open-source and fine-tuned models, such as Llama 2 70B, enhancing productivity in developing LLM workflows. Users can easily switch between model providers in the LangSmith Playground and test different prompts and models without any coding, thereby streamlining the otherwise iterative and time-consuming process of prompt optimization.