April 2024 Summaries

At Braintrust, automated evaluations are being used by AI teams to improve the development speed of their applications. Prior to this, teams relied on manual review and benchmarks, which fell short in scaling and application specificity. Automated evaluations offer a high-leverage way for teams to quickly understand product performance, identify regressions, and improve their dev loop. Three approaches to automated evaluations are discussed: LLM evaluators, heuristics, and comparative evals. These methods enable teams to set up basic structure around automated evaluations, unlocking the ability for developers to start iterating quickly and making human review time much higher leverage.

In the field of AI engineering, establishing a set of real-world examples, known as "evals", is crucial to understand how changes will impact end users. However, finding great eval data, identifying interesting cases in production, and tying user feedback to evals can be challenging problems. To overcome these issues, connecting real-world log data to evals allows for the evaluation of new and interesting cases in the wild, enabling improvements and avoiding regression. This is achieved by structuring evals as a function of data, prompts/code, and scoring functions, and utilizing tools like Braintrust's Eval function that streamlines this process. By capturing and utilizing logs, teams can power their evals with real-world examples, making it easier to identify interesting cases and improve AI products. As teams scale, filtering logs to only consider the most interesting ones becomes critical, and using filters, tracking user feedback, or running online scores can help uncover test cases that need improvement. Braintrust's solution provides a unified UI for exploring logs and evals, automating code reuse, and storing datasets in a cloud environment.

We are excited to announce that Braintrust was selected to be in the Enterprise Tech 30, a list of private companies in enterprise technology with significant potential to transform how tech enterprises operate. We support many incredible AI teams at top companies such as Zapier, Coda, and Notion, and we're eager to help more companies like them by shipping AI features. If you're interested in prioritizing AI feature development or would like to explore job opportunities, sign up for free or get in touch with us. The full list of Enterprise Tech 30 companies can be found here.