April 2025 Summaries

Bryan Cox and Ankur Goyal hosted a webinar titled "In the Loop: Technical Q&A" where they discussed effective eval best practices. Bryan emphasized keeping evals simple, starting with around 10 examples, and building a feedback loop to iterate quickly. They also introduced several common scoring functions teams start with, including Levenshtein distance, factuality, and closed QA. Additionally, they discussed Braintrust's approach to multi-step prompt chaining, integration with continuous integration, user feedback handling, multimodal data evaluation, balancing automated scoring with human review, Brainstore logging database, and the potential use of synthetic data in evals. The future of evals will involve automation of more tasks, aligning AI outputs with human expectations, and involving more team members beyond just engineers.

Bryan Cox and Ankur Goyal hosted a webinar titled "In the Loop: Technical Q&A," focusing on evaluation methods, agents, and observability in machine learning workflows. The session covered starting with simple evaluation metrics such as Levenshtein distance and factuality prompts, integrating evaluations into continuous integration systems using Braintrust's GitHub actions, and handling user feedback while ensuring privacy through anonymization features. It also introduced Braintrust's new agents feature for multi-step prompt chaining, support for multimodal data evaluations, and emphasized balancing automated scoring with human review. Brainstore, Braintrust’s logging database, was highlighted for its ability to manage large-scale LLM workloads efficiently. The discussion included the role of synthetic data as a complement to real data and the potential for evaluations to automate and align AI outputs more effectively with human expectations.

Braintrust is an AI observability platform designed to enhance stability without compromising the core logic of LLM infrastructure. It provides non-blocking logging by default, ensuring that log data is sent in the background without blocking running code. The SDK logger is initialized with asyncFlush set to true, allowing for efficient and asynchronous logging. Braintrust is built for serverless environments, accounting for unique constraints such as ephemeral functions and edge runtimes. It offers tunable parameters to prevent log operations from holding up functions and provides a reliable proxy service that routes requests through AWS CloudFront. The platform also features resilient prompt management, enabling versioning and iteration of LLM workloads while maintaining production reliability. By designing observability by default, Braintrust aims to enhance stability without introducing new points of failure, providing confidence through its design.