October 2024 Summaries

You can now upload, download, and contextually view attachments in Braintrust, expanding the capabilities of AI applications that accept and generate images, audio, video, and more. This feature allows users to log various types of binary attachments, including images, audio files, videos, and PDFs, which can be securely stored and accessed by the uploading user's organization. The attachment feature simplifies debugging and provides observability for multimodal and complex applications, enabling users to view and download attachments in the Braintrust UI.

You built an AI application, curated test examples, picked a scoring function, and ran an evaluation. Now you need to think about that score and what steps to take next to continuously improve both your AI application and evaluation process. To determine where to focus first, review 5-10 actual examples from your evaluations, inspecting the trace for each row, input, output, and scoring outcome. Analyze these examples to identify patterns around how your application is performing and whether your scoring is accurate. You can then decide whether to refine your evaluations or make changes to your application, iterating through both with rapid development loops to continuously improve performance.

Notion, a popular workspace platform, has developed world-class AI features by integrating generative AI models such as GPT-4 into its product. Notion's commitment to customer experience led them to invest in AI development, resulting in innovative features like Notion AI, which powers capabilities for searching workspaces, generating documents, analyzing PDFs, and answering questions. However, the team realized their existing workflows for evaluating AI products were not up to the challenge, prompting a partnership with Braintrust to dramatically improve their evaluation workflow, enabling faster iterations and higher quality features in production. With Braintrust, Notion has transformed its eval workflow, allowing it to triage and fix 30 issues per day compared to just 3 previously, setting a new standard for AI product teams to evaluate and improve their generative AI products.

Functions in Braintrust are flexible AI engineering primitives that allow developers to build and scale AI-powered products with ease. They enable the creation of atomic, reusable building blocks for executing AI-related logic, including prompts, tools, and custom scorers. Functions can be created in code or the UI, linked together, and invoked through APIs. This enables the development of agentic applications quickly and efficiently, while maintaining security and scalability. The functions are open and embrace standards, allowing them to work with various models and frameworks. They represent a significant shift in the software engineering lifecycle, bringing together evaluation, logging, datasets, prompts, and the playground into one powerful workflow.

Today, we're thrilled to announce that we've raised $36 million to advance the future of AI software engineering, bringing our total funding to $45 million. We've learned key lessons through working with top AI engineering and product teams from leading companies including Notion, Stripe, Vercel, Airtable, Instacart, Zapier, Coda, The Browser Company, and many others about crafting effective prompts, the importance of evaluations, and the value of production logs in improving quality over time. This funding round was led by Martin Casado at Andreessen Horowitz with participation from existing investors and new backers including Datadog, Databricks Ventures, Guillermo Rauch, Simon Last, Bryan Helmig, Greg Brockman, and Arthur Mensch. With this support, we'll double down on solving challenges in building robust LLM applications, driven by collaboration between product engineers, designers, and product managers who continuously experiment with new prompts. We're also excited to introduce functions, a powerful new feature that allows developers to create tools, prompts, and scorers directly in their codebase and upload them to Braintrust with ease, unlocking a seamless developer experience.