May 2023 Summaries

Jacob Lee explores integrating large language models (LLMs) such as GPT-4 and Claude into client workflows beyond chat interfaces, focusing on creating a pipeline where LLMs interact with traditional APIs. He identifies challenges in using natural language outputs with existing APIs, especially when formatting data for applications like Airtable. Lee experimented with custom prompting strategies to improve output formatting but found them tedious and unreliable. He discovered a solution using LangChainJS, an open-source framework that includes output fixing parsers powered by TypeScript's Zod validation framework and LLMs to handle poorly formatted outputs. The parsers provide consistent formatting instructions and enable clean data outputs, which can be directly integrated into APIs like Airtable. This approach eliminates the need for complex string parsing and enhances project efficiency. Lee's enthusiasm for the technology led him to join the LangChain team, where he plans to continue sharing insights and developments in LLM applications.

Lance Martin discusses the potential of retriever-less architectures in LLM question answering (Q+A) due to the emergence of models with significantly larger context windows, such as Anthropic's 100k token model, which can handle entire documents without a separate retrieval step. This development raises questions about the necessity of traditional retrieval methods, especially for smaller document sets. Testing reveals that while the Anthropic 100k model performs well against retrieval-based methods like kNN and SVM in some cases, it suffers from higher latency and occasionally less accurate answers due to the inability to inspect retrieved chunks. Despite these challenges, retriever-less architectures offer simplicity and are promising for applications with manageable document sizes and where latency is less critical, as LLM context windows continue to expand and models become more efficient.

Willem Pienaar and Shahram Anver discuss the growing concern over prompt injection (PI) attacks on applications built using Language Learning Models (LLMs), highlighting how such attacks can manipulate outputs, expose sensitive data, and enable unauthorized actions. Rebuff, an open-source framework, offers a solution by providing a self-hardening detection mechanism against these attacks, utilizing multiple defense layers such as heuristics, LLM-based detection, vector databases, and canary tokens. The authors demonstrate how Rebuff can be integrated into applications, showing its ability to detect potential SQL injection attacks through an example scenario. Despite its efficacy, Rebuff is still in its alpha stage and comes with limitations, including the potential for false positives and negatives and the need for ongoing development and community involvement to enhance its robustness against skilled attackers.

LangChain is introducing a new type of agent executor called "Plan-and-Execute," which is designed to address the limitations of existing "Action" agents by separating high-level planning from short-term execution. Inspired by concepts from BabyAGI and the Plan-and-Solve paper, this new framework aims to handle complex long-term planning more effectively, albeit with increased calls to language models. The Plan-and-Execute agents utilize a planner, typically a language model, to strategize steps and an executor to implement them, allowing for improved reliability and flexibility. While the approach involves more calls, it promises future enhancements like better support for long sequences of steps, revisiting plans, and rigorous evaluation methods, drawing inspiration from existing work to refine the system.

The text explores the potential integration of feature stores with language model applications, specifically focusing on how feature stores can enhance prompt construction by incorporating real-time, user-specific data. In traditional machine learning, feature stores centralize and serve engineered features to models, a concept that could be adapted to provide personalized prompts for language models, despite many applications currently relying on pre-trained large language models (LLMs) rather than training from scratch. The discussion highlights various prompt construction strategies, from hard-coded strings to those incorporating user input and external data, suggesting that feature stores can enrich these prompts by providing complex, real-time information. This approach is demonstrated using examples from feature stores like Feast, Tecton, and FeatureForm, indicating a future where language model applications leverage real-time data to offer more personalized and contextually aware experiences, such as chatbots with real-time awareness, personalized marketing content, and tailored recommendations.

LangChain has introduced a new abstraction for more complex tool usage, allowing tools to accept multiple inputs of varying types, and has developed a new agent class compatible with these tools. Initially, LangChain's tool use was limited to single-string inputs per tool due to model constraints, but advancements in language models like GPT-4 have enabled more sophisticated interactions. The new "multi-action" agent framework enables agents to execute multiple actions, and the introduction of "Structured Tools" allows for complex interactions between language models and tools by wrapping functions that agents can interface with easily. These tools are defined by a unique name, description, arguments schema, and run functions, and they support various operations from arithmetic to API requests. LangChain has also released new tools for file management and web browsing, alongside guidance for implementing custom structured tools. The new tools are compatible with existing agents if they accept a single string argument, while more complex structured tools require customization for full compatibility with older agents.

LangChain has announced enhancements to their callbacks system, which is crucial for tasks such as logging, tracing, and integrating third-party applications, aimed at improving concurrent runs and request-specific callback handling. Initially designed for non-async Python applications, LangChain now supports asyncio Python and JavaScript/TypeScript, necessitating better abstractions for handling multiple concurrent runs. The new system allows developers to declare callbacks in constructor arguments or directly in methods, ensuring callbacks are scoped appropriately for specific runs, which simplifies the process and enhances reusability by eliminating the need to attach callback managers to all nested objects. The changes also introduce a runManager second argument to async methods, facilitating logging for custom chains, and a context manager for easier tracing. Notably, global callbacks and the global tracer are deprecated in the latest versions, with the verbose flag now limited to controlling stdout and console callbacks only. The improvements draw inspiration from the Python logging module and web server frameworks like Express, opting for explicit function arguments to ensure compatibility and ease of debugging across platforms.

Lance Martin's blog post discusses the release of an open-source auto-evaluator tool designed to improve the quality of question-answering (QA) systems using large language models (LLMs). The tool, available as a free hosted app and API, evaluates the quality of QA chains by auto-generating and grading test sets for given input documents. It addresses common issues such as hallucination and poor answer quality by allowing users to experiment with different QA chain configurations and components. Inspired by recent work from Anthropic and OpenAI, the auto-evaluator combines model-written and model-graded evaluations in a single workspace, facilitating modular testing with LangChain's abstraction. The app supports various retriever approaches, such as k-Nearest Neighbor, SVMs, and TF-IDF, and highlights areas for improvement, including file handling, prompt refinement, and model selection. The post encourages contributions to the open-source project, particularly in enhancing file transfer efficiency, refining prompts for model-graded evaluations, and exploring additional retriever options.