May 2024 Summaries

Zapier, a leading workflow automation platform, has successfully built production-ready AI products by following a structured development process. Initially, they quickly validate their AI feature idea using existing models and then build and ship v1 with sub-50% accuracy. After collecting user feedback, they establish a set of evaluations to accurately score their application on diverse examples. The team iterates to improve product quality, expands the product's availability and capabilities as accuracy increases, takes pride in shipping an amazing AI product, and optimizes cost and latency.

The text discusses the challenges of developing software with AI and proposes three core development loops: Exploration, Evaluation, and Data Collection. The Exploration loop enables rapid iteration by quickly testing changes in a low-stakes environment, while the Evaluation loop provides high certainty about the impact of a change through automated testing and scoring. The Data Collection loop builds high-quality datasets for testing by identifying patterns and interesting results in production logs and capturing inputs to generate signal in the other two loops. Braintrust provides tools that support these three loops, enabling fast and reliable development with AI.