How to turn LLM production failures into regression tests

LLM production failures often appear successful in observability tools because they may not trigger exceptions, despite incorrect user-facing answers. To address this, Braintrust provides a system to capture failed traces, label failure modes, and turn them into regression tests for future releases. It ensures that each diagnosed failure, such as hallucinations, retrieval misses, tool-call errors, or format violations, is preserved in a dataset and evaluated using custom scorers both in continuous integration (CI) and on live traffic. This approach highlights the importance of using production traces as the source of truth, allowing engineering teams to convert real-world failures into durable regression tests. By incorporating these traces into regression datasets, Braintrust allows for the detection and prevention of recurring failure patterns, enhancing the reliability and accuracy of LLM systems. The process involves capturing production traces with sufficient context, diagnosing failure modes, promoting traces into datasets, writing appropriate scorers, and integrating them into CI/CD workflows for ongoing evaluation and release gating.