November 2024 Summaries

The text discusses the challenges and solutions associated with evaluating autonomous AI systems, particularly in multi-turn interactions and complex workflows. It highlights that traditional testing and manual reviews are inadequate for capturing multi-step failures in AI agents, necessitating a systematic approach to agent evaluation. The text introduces Braintrust, a comprehensive platform offering features like Loop for creating custom scorers from natural language descriptions, remote evaluations for no-code testing, and AI-powered log analysis to identify failure patterns. Braintrust's unified platform integrates evaluation, observability, and optimization, reducing tooling fragmentation and accelerating iteration cycles. It contrasts Braintrust's capabilities with other platforms like LangSmith, Vellum, Maxim AI, and Langfuse, emphasizing Braintrust's production-grade features, ease of use, and the potential for significant accuracy improvements and faster development cycles. The text explains that effective agent evaluation involves assessing decision-making, tool selection, and output quality across interactions, and it positions Braintrust as a leading solution for teams needing framework-agnostic evaluation with deep observability and streamlined scorer creation.

Building a Retrieval Augmented Generation (RAG) app with MongoDB Atlas, Braintrust simplifies the workflow of iterating on RAG applications by pushing the retrieval tool from the codebase to the UI, allowing users to experiment with different prompts and models side-by-side. The application is defined as two components: a system prompt containing instructions for retrieving content and synthesizing answers, and a vector search tool implemented in TypeScript that embeds a query, searches for relevant documents, and returns them. To get started, users need to create a new cluster in their Atlas account, upload vectors from the Braintrust documentation, define the retrieval tool, push it to Braintrust, and then use the UI to try out the tool and tweak the prompt to improve results. The playground provides an interactive environment for testing out the prompt and tool call, allowing users to evaluate the results using a custom scorer and create a more detailed analysis by running full experiments.

Gemini models have been evaluated for their vision capabilities, including document extraction. The results show that Gemini models use significantly fewer tokens per image compared to GPT-4o models, are faster at processing inputs, and slightly more accurate in factuality. However, they generate more completion tokens than GPT-4o models. These findings suggest that Gemini models have potential advantages over GPT-4o models for certain vision tasks. The AI proxy allows users to easily integrate Gemini into their applications with a single-line code change, making it easy to experiment with the model's multimodal capabilities and fine-tune prompts to meet specific needs.

Support for Python tool functions has been announced by Braintrust, a platform that enables developers to create custom code blocks called "tools" that can be invoked by large language models (LLMs). These tools are defined in code and synced with the UI using `braintrust push`, which also bundles and uploads any required libraries or dependencies. To address challenges in compiling dependencies, Braintrust leveraged the uv Python package manager to create a cross-bundling virtual environment for each tool, making it easy to deploy reusable and composable tools in both TypeScript and Python. With this support, developers can now build simple agents that perform tasks like web-scraping or API execution, and deploy them across the UI and API using prompts.

The OpenAI Realtime API is designed for building advanced multimodal conversational experiences, but it has three key pain points that need to be solved before its use in production applications: user-facing credentials, logging, and evaluations. To address these challenges, Braintrust has released an AI proxy that securely manages the OpenAI API key, issuing temporary credentials to the backend and frontend, thereby offloading infrastructure burden and allowing developers to focus on building their app. The proxy also provides logging capabilities, enabling developers to view logs for LLM calls and multimodal content in a unified format.