June 2026 Summaries

In a benchmark comparing the performance of GLM-5.2 from Z.ai and Anthropic's Opus 4.8, GLM-5.2 demonstrated notable cost efficiency in long-context retrieval for coding agents, despite Opus 4.8 maintaining a slight edge in accuracy. Evaluated in collaboration with Baseten, GLM-5.2 was tested under real-world production constraints using mechanically extracted questions from the CPython standard library, revealing that GLM-5.2 offers significant cost savings—approximately 76-78% lower provider cost per trace—while maintaining competitive performance. The evaluation highlighted GLM-5.2's ability to preserve retrieval accuracy across context sizes of 25K and 50K tokens, making it a viable choice for high-volume, cost-sensitive applications, despite its sensitivity to latency under load. The study emphasizes the importance of serving configuration in optimizing performance, with Baseten's platform offering control over deployment parameters to mitigate latency spikes. These findings underscore GLM-5.2's potential in enterprise contexts where long-context retrieval is crucial, as it effectively balances cost and performance, making it a strategic choice for applications like code intelligence, financial document analysis, and medical record summarization.

The technical report evaluates the performance and efficiency of long-context language models, specifically GLM-5.2 and Opus 4.8, using the RULER benchmark to determine their capability in retrieving exact facts from large contexts without relying on memorized knowledge. Despite claims of handling large token windows, many models experience significant performance drops as context length increases. The study highlights the importance of models attending to the correct part of the prompt and serving systems efficiently managing long prefixes. GLM-5.2 employs sparse-attention architectures and content-dependent indexing to manage long-context computations, incorporating techniques like IndexCache to improve efficiency. The evaluation uses CPython's standard library as a testbed due to its determinism and structural richness, allowing for machine-checkable ground truth based on AST-derived questions. The findings reveal that Opus 4.8 outperforms GLM-5.2 in terms of retrieval quality but at a higher cost, while GLM-5.2 is noted for its cost-effectiveness and competitive performance in exact long-context retrieval. The study stresses the significance of infrastructure in achieving stable latency and cost-effectiveness, with GLM-5.2 showing potential for fast responses under optimal conditions.

Braintrust for Startups is a newly launched program aimed at providing early-stage teams with the infrastructure needed for effective agent observability and evaluation, which many startups neglect due to resource constraints. This initiative offers these startups access to the same platform utilized by leading AI companies, integrating evaluation and observability to enhance agent performance and production understanding. Participants receive six to twelve months of access to the Braintrust Pro plan, which includes features such as custom charts and priority support, along with opportunities for networking and learning through exclusive events. The program is supported by prominent VC firms and is open to startups with at least $100k in funding, with extended access available to those associated with partner firms at Series A or earlier.

Open-source models have significantly advanced in performing tasks crucial for coding agents, such as reading long contexts and retrieving precise information from extensive codebases, at a fraction of the cost compared to frontier models. GLM-5.2, one of the leading open-source models, is available on Braintrust, where users can compare its performance with existing models using their own prompts and traces until July 31. In a comparative evaluation with Claude Opus 4.8, GLM-5.2 showed slightly less accuracy but was substantially more cost-effective, performing at about one-sixth of the cost per correct answer. Braintrust offers a streamlined experience for running GLM-5.2, eliminating the need for a separate inference setup, thereby allowing users to evaluate the model's quality and production behavior efficiently. This model is particularly beneficial for high-volume agent workflows, where cost savings may outweigh minor differences in accuracy.

In complex Language Model (LLM) applications, logging alone can be insufficient for diagnosing issues because it captures independent events without illustrating the interconnected steps of a request, which is essential for understanding why a particular run failed. Logs provide discrete, timestamped records of events such as exceptions or infrastructure changes, which are useful for standalone analyses like error reporting or compliance audits. However, they fall short in multi-step LLM processes where tracing becomes crucial. Tracing records the entire path of a request as a series of connected spans, allowing teams to see how each step, from retrieval to final response, transpired and contributed to the outcome. This visibility is particularly valuable in LLM applications where non-deterministic behavior and multi-step executions can lead to different results with the same input. By integrating both logging and tracing, along with tools like Braintrust and OpenTelemetry, teams can achieve comprehensive observability, enabling them to debug issues effectively and evaluate production behavior by connecting structured logs with traces to provide a complete picture of request-level dynamics.

Agent tracing provides an in-depth view of how a support agent's execution path unfolds during a run, capturing each step's inputs, outputs, and the sequence of decisions leading to the final response. This method proves essential in debugging production failures, such as when a retrieval step returns an outdated policy document that leads to incorrect answers. Unlike traditional logging or LLM tracing, agent tracing offers a structured parent-child span tree that connects model calls, tool calls, retrievals, and state changes, providing comprehensive visibility into the agent's decision-making process. This detailed trace enables teams to pinpoint the exact step causing a failure, facilitating efficient debugging and forming the basis for creating regression evaluation cases to prevent future errors. Braintrust's agent tracing helps transform production failures into permanent evaluation cases, allowing teams to test changes to prompts, models, retrievals, or tools before they reach production, thereby enhancing the reliability and accuracy of AI-driven agents.

An AI gateway acts as a control layer through which all large language model (LLM) traffic passes before reaching a model provider, unifying access controls, quotas, cost tracking, and audit logging across providers. This facilitates production governance and developer routing by offering model routing and a unified API within the same core layer. AI gateways are categorized into infrastructure-first gateways, which extend existing API management systems to AI traffic, and LLM-native gateways, which focus on model access and provider abstraction. Key criteria for selecting an AI gateway include provider and model breadth, rate limiting, access control, governance, caching, cost tracking, audit logging, and actionable observability. Several AI gateways, such as Braintrust, Portkey, LiteLLM, Kong AI Gateway, SUSE AI Universal Proxy, and Cloudflare AI Gateway, offer varying capabilities depending on team needs, from production governance and observability to infrastructure control and edge caching. Braintrust Gateway stands out by integrating routed model requests into the release control process, enhancing evaluation workflows, and offering comprehensive logging and tracing capabilities, making it a preferred choice for teams focusing on production AI applications.

Running an AI agent in production involves processing raw traces that lack straightforward answers, requiring careful querying and evaluation to discern patterns and insights. The analysis of 1,781 agent traces from Exgentic, hosted on Hugging Face, revealed that the choice of harness significantly impacts performance, being approximately seven times more influential than the model itself. Changing the harness can drastically shift success rates from 12% to 92% without affecting token costs. Open-weight models like DeepSeek and Kimi have proven to be production-ready, particularly for coding tasks, achieving success rates comparable to closed models but with the advantage of self-hosting capabilities. The cost per task and cost per success can vary greatly, with open-weight models offering more cost-effective solutions for coding tasks, while closed models like GPT-4.1 are more cost-efficient for conversational tasks. High average performance does not guarantee reliability, as some configurations may perform well overall but fail in specific tasks. There is no universally best model, as different models excel in different task types, and the evaluation highlights the importance of considering both success rates and cost efficiency in selecting AI configurations. Additionally, failure patterns differ between coding and conversational tasks, with coding failures often involving excessive token usage and conversational failures resulting from premature task abandonment. The use of Braintrust enabled structured analysis and regression, turning raw traces into actionable insights and facilitating targeted experiments to address specific failure modes.

Stateful agents, unlike traditional prompt-response models, retain memory across multiple steps and interactions, enabling them to operate effectively in unstructured and dynamic environments. These agents accumulate context, remember prior actions, and make decisions based on the state of external systems, which can change independently over time. This complexity introduces unique challenges for evaluation, as traditional methods focusing on single-step prompts are inadequate. Proper evaluation requires capturing the agent's state and trajectory over time, including the potential side effects of their actions. Effective stateful evaluations should involve real interaction with systems to capture meaningful state changes, turning real-world failures into test cases for continuous improvement. The process involves observing agent behavior through detailed logging, transforming failures into actionable datasets, and leveraging automated scoring within CI/CD pipelines to prevent regressions. Tools like Braintrust facilitate this by providing observability, scoring mechanisms, and clustering of failure patterns, allowing teams to iteratively refine and ensure the reliability of stateful agents in production environments.

A unified LLM API streamlines access to multiple language model providers like OpenAI, Anthropic, and Google by routing requests through a single integration point, eliminating the need for separate SDKs, authentication, and billing systems. This setup allows developers to easily switch between models using parameters, simplifying the process of adding or testing new models without altering existing client setups. The choice of a unified LLM API provider often hinges on factors such as compatibility with OpenAI SDK, provider and model breadth, caching capabilities, cost visibility, observability, and the level of control over deployment and vendor lock-in. Notably, Braintrust Gateway is highlighted as a leading provider, offering extensive model access, evaluation, and debugging tools, making it suitable for production AI teams that require robust model management and quality control. Other providers like OpenRouter, Vercel AI Gateway, LiteLLM, Portkey, Together AI, and Groq cater to different needs, ranging from broad model access and governance controls to low-latency inference and open-model support.

In the context of AI-driven customer support systems, the focus should be on developing cost-effective control logic that ensures production-safe outcomes rather than solely choosing the cheapest models. This approach involves optimizing tool selection, model routing, retries, fallbacks, and safety-based escalations to balance quality and cost effectively. Evaluations show that strategies like "Retry, then escalate" and "Smart escalation" outperform cheaper, single-model approaches by resolving a higher percentage of tickets at a lower cost per resolved ticket. The effectiveness of these strategies is attributed to their ability to adapt to the complexities of real-world scenarios through dynamic routing and escalation, as opposed to static or purely cost-driven models that may fail to maintain quality. Tools like Braintrust provide the necessary infrastructure to log workflows, define evaluations, and compare strategies, enabling organizations to experiment with and refine their control logic to ensure efficient and high-quality customer support.

The text outlines a methodology for identifying hidden failure patterns in AI agents' production traffic using clustering techniques facilitated by Braintrust Topics. It explains that traditional scorers and alerts often miss failures that aren't predefined, such as stale context or tool retries, due to their reliance on known failure descriptions. The approach involves clustering production traces by behavior, reviewing unusual clusters to confirm failure patterns, and operationalizing confirmed patterns into scorers, evaluation datasets, or review workflows. Braintrust Topics automates the clustering of production traces, categorizing them into behavior clusters that help teams discover failure modes that standard monitoring might overlook. The guide emphasizes the importance of validating cluster findings through a detailed review of traces and understanding user sentiment to prioritize issues that impact user experience. It also discusses the use of custom facets for domain-specific failure tracking and highlights the iterative nature of failure discovery as agents and production environments evolve.

AI agent production traffic provides valuable insights into user behavior and unmet needs by classifying interactions based on intent, sentiment, and product-specific signals. This approach allows product teams to derive evidence for roadmap decisions directly from real user interactions, bypassing traditional methods like interviews and surveys, which may not capture the full spectrum of user experiences. Braintrust Topics facilitates this process by automatically classifying agent traffic into tasks, sentiment, and issues, enabling product managers to identify trends and prioritize potential improvements or new features. The tool offers various views, such as scatterplots and lists, to visualize and analyze traffic patterns, helping teams to focus on significant clusters that indicate capability gaps, feature requests, or capability discoveries. By supplementing these insights with custom facets tailored to specific business needs, product teams can integrate comprehensive traffic analysis into their strategic planning, ensuring that roadmap decisions are grounded in authentic user behavior and not solely on selective qualitative research.

AI agents generate vast amounts of production traces, but creating effective evaluation datasets from these logs is challenging due to the randomness of sampling and the limitations of hand-written tests. By analyzing production usage patterns, particularly focusing on tasks, user sentiment, and issues, product and engineering teams can identify significant workflows, user frustrations, and recurrent failures to construct more representative evaluation datasets. Braintrust Topics facilitates this process by clustering agent traces based on task intent, sentiment, and issue type, enabling teams to convert high-value trace groups into evaluation datasets that mirror real-world agent interaction. This approach ensures better coverage of potential production risks, unlike random sampling which may overlook rare but impactful failure modes. The system allows for continuous updates and alignment with current production behavior by regenerating patterns daily, and it supports the use of custom facets to address domain-specific questions. The ultimate goal is to create evaluation suites that are manageable in size but comprehensive in risk coverage, thereby improving the reliability and user experience of AI agents.

LLM model routing serves as the operational layer in AI systems that determines which model or provider processes each request, with its effectiveness varying based on whether it prioritizes resilience, cost control, or quality-based model selection. Different platforms offer various focuses: some emphasize resilience through backup systems and provider ordering, others prioritize cost-efficiency by routing simpler requests to cheaper models, and the most robust setups link routing decisions to actual performance data from live traffic. Braintrust stands out as a leading platform for teams that desire model selection informed by evaluation, online scoring, and real-time traffic experiments within a cohesive workflow. It provides a unified API for accessing models from major providers like OpenAI, Anthropic, and Google, supporting both custom and self-hosted models while offering tools for caching, credential management, and routing observability. Other notable platforms such as OpenRouter, Vercel AI Gateway, Portkey, and LiteLLM offer varying strengths, from broad model access and managed provider routing to open-source, self-hosted solutions, each catering to specific operational needs and priorities.

Braintrust is a versatile tool designed to accommodate the diverse and evolving stacks of AI teams, providing a seamless way to instrument applications for consistent tracing, evaluation, and debugging across various frameworks and model providers. It supports auto-instrumentation for multiple languages, capturing AI library calls without requiring individual client wrapping, and offers manual and OpenTelemetry instrumentation for more control. Braintrust enables teams to run evaluations using its Eval primitives, which can be applied across different frameworks and providers, allowing for consistent scoring and debugging processes. It facilitates multi-provider use by standardizing observability and evaluations, offering a gateway for API compatibility across providers, and supporting custom models alongside standard ones. The platform integrates with existing observability tools and provides options for data import and synchronization, making it adaptable to different development environments and workflows. By ensuring that tracing and evaluation are framework-neutral, Braintrust allows teams to maintain consistent quality assessment and debugging practices, regardless of the underlying technology stack.

AI agent analytics tools are designed to help engineering and product teams understand and improve the behavior of AI agents in production by classifying, grouping, and analyzing conversation patterns. These tools address challenges such as recurring user intents, negative sentiment, and failed tool calls by providing insights into production traffic and enabling trace review of underlying issues. Braintrust is highlighted as a leading choice due to its comprehensive Topics analytics layer, which automatically classifies agent behavior across tasks, sentiment, and issues, and links these classifications to evaluation workflows, making it particularly effective for release control. Other tools like Datadog, Langfuse, Galileo, and HoneyHive offer varying features, from infrastructure monitoring to open-source observability and runtime guardrails, catering to different organizational needs. The guide emphasizes the importance of automatic classification, multi-dimensional analysis, and the ability to integrate evaluation workflows for effective AI agent analytics, with Braintrust providing a seamless connection from identified patterns to production control and release gates.

AI conversation analytics is a practice that involves classifying and grouping every interaction an AI agent has with users to identify trends and recurring patterns in production traffic. This process aims to improve the understanding of user tasks, sentiment, and failure patterns, which aids in prioritizing issues and converting repeated failures into evaluation cases before subsequent releases. Braintrust, a leading platform in this area, utilizes its Topics feature to automatically classify conversations by task, sentiment, and issues without requiring predefined classifiers, transforming production conversations into structured datasets for evaluation. This capability allows teams to filter logs, inspect source conversations, and convert recurring patterns into datasets, online scorers, and review queues, thereby enhancing the quality and reliability of AI systems. Other tools like Galileo, HoneyHive, Datadog, and Langfuse offer varying features focused on observability, tracing, and evaluation but may require more setup or focus on different aspects like runtime protection or open-source deployment.

Continuous evaluation is a critical process for ensuring the quality and reliability of AI agents after they are deployed in production environments. Unlike pre-deployment or CI evaluations, which rely on predefined test cases, continuous evaluation automatically scores live production traces based on classifications like Task, Sentiment, and Issues using BraintrustTopics. This approach helps identify failures and edge cases that were not anticipated during initial testing phases. By utilizing a system of classifiers and predicates, production failures can be consistently monitored without the need for a scorer per failure mode, thereby reducing maintenance complexity. Scored traces can trigger alerts, initiate review processes, or be promoted to regression tests, ensuring that production failures contribute to ongoing quality improvement. The process is designed to be scalable, with adjustable sampling rates to manage scorer load and costs, and it supports the integration of both built-in and custom facets for more tailored evaluations. This continuous evaluation framework allows for proactive monitoring and faster response to issues, ultimately enhancing the robustness of AI agents in real-world applications.

Hallucination evaluation in AI systems is a comprehensive process that assesses whether outputs are factually incorrect, unsupported by retrieved sources, or inconsistent with previous context. Effective evaluation requires a combination of suitable metrics, scoring methods, and reference sources to align with potential failure modes in production, whether those involve retrieved documents, curated datasets, or repeated outputs without a reference. Metrics like groundedness, faithfulness, factuality, and consistency each address different aspects of hallucination, necessitating separate checks for each failure mode and highlighting the importance of choosing the right evaluation setup. Braintrust offers an integrated workflow that combines built-in scorers, custom scorers, and human review to transition hallucination checks from measurement to release control, supporting tools like LLM judges, fine-tuned models, and semantic entropy for varied evaluation needs. By utilizing human reviews for calibration and employing methods like consistency sampling and semantic entropy, evaluations can be tailored to specific tasks, ensuring that automated scoring aligns with real-world user risks and improving system reliability over time.

Braintrust's Topics is an innovative solution designed to intelligently analyze production logs at scale, addressing the challenge of extracting valuable insights from vast amounts of complex agent traces. It introduces an architecture that uses a summarization-first approach, where a language model generates concise summaries of traces that are then embedded, clustered, and classified, allowing for cost-effective and continuous processing. The architecture leverages concepts from Anthropic's Clio paper and avoids embedding raw data, focusing instead on embedding these summaries to maintain tractability and reduce costs. Topics supports various dimensions like task, sentiment, and custom categories without requiring multiple pipelines, thereby streamlining the analysis process. Automation ensures that the system runs continuously, updating topic maps regularly, while maintaining stable cluster identities, allowing for consistent trend analysis and queryable outputs. The system's design emphasizes developer experience by providing a seamless integration that requires minimal intervention while enabling custom adjustments, making it a robust and scalable solution for understanding logs in production environments.

LLM cost tracking becomes problematic when relying solely on provider invoices, as these do not explain the specifics of spending increases, such as which customer or feature caused it. To address this, request-level attribution is suggested, where every model call includes metadata linking costs to users, features, and workflows, facilitating better decision-making regarding product, pricing, reliability, and margins. The text outlines methods for connecting LLM spend to request metadata, creating cost rollups, normalizing spend across providers, and implementing alerts, caps, and kill switches to manage costs effectively. It emphasizes the importance of tagging requests at the call site to maintain context and prevent the need for post-hoc log parsing. Moreover, the Braintrust platform is highlighted for its ability to connect cost, traces, eval scores, and release gates, enabling cost reductions to be evaluated for their impact on quality and latency before production implementation. The document further discusses enforcing cost limits through alerts and caps, ensuring cost-saving changes do not degrade output quality, and making informed decisions on cheaper model deployments without compromising production thresholds.

The text delves into the intricacies of token usage in large language models (LLMs) and the factors contributing to increased token consumption, such as prompt bloat, context window pressure, and agent loops. It emphasizes the importance of structured token tracking across multiple levels—prompt and completion tokens per call, context window utilization, and span-level tracking within agent traces—to identify and resolve production issues effectively. The guide explores methods for logging token usage through integrations like BraintrustSDK, OpenTelemetry, and auto-instrumentation, which connect token counts to various operational metrics, making it easier to control costs and enhance performance. It also highlights the need for detailed token attribution in agent workflows and monitoring context window utilization to prevent overflow errors. Additionally, it discusses the impact of caching and batching on token usage and underscores the importance of avoiding common token tracking errors to ensure accurate data for decision-making.

Braintrust Topics provides AI teams with an automated and scalable method for reviewing production traces by classifying logs based on user intent, sentiment, and issues, ultimately clustering similar traces for more efficient analysis. This approach addresses the challenge of manually reviewing the vast number of logs generated by AI agents, which can reach tens of thousands per week. By automating trace classification, Braintrust Topics enables teams to quickly identify recurring problems, build evaluation datasets from real production traffic, and prioritize which patterns need further review or scoring. The system operates through a daily pipeline encompassing preprocessing, facet summarization, clustering, and classification stages, with built-in facets focusing on Task, Sentiment, and Issues. Custom facets can also be created for more domain-specific inquiries. Braintrust Topics integrates into the broader Braintrust evaluation workflow, enhancing trace observability and providing actionable insights from production data. It is available across different plan tiers, with specific conditions for enabling and utilizing its features, making it a valuable tool for teams aiming to improve AI agent performance and user experience.

AI governance for LLM applications encompasses evaluating model outputs, recording production behavior, controlling access, and enforcing policies across development and production phases. Effective governance requires eval-time scoring, production audit, access control, and runtime enforcement to ensure compliance and mitigate risks. Braintrust emerges as a leading AI governance platform, offering a unified workflow that integrates evaluation, production tracing, offline evaluation, CI release gating, and human review, all while maintaining audit-grade tracing, role-based access control (RBAC), and compliance certifications such as SOC 2 Type II and HIPAA. While Braintrust excels in evaluation and audit capabilities, other platforms like Galileo focus on low-latency runtime protection, Credo AI on portfolio-level governance, Fiddler AI on classical ML monitoring combined with LLM oversight, and Patronus AI on regulated-domain evaluator coverage. Braintrust's deployment flexibility, including hybrid and self-hosted options, ensures data residency and security, making it suitable for regulated industries seeking comprehensive governance solutions for LLM applications.

Braintrust offers a streamlined approach to incorporating LLM observability into AI applications via its CLI, which simplifies the process by handling authentication, SDK setup, instrumentation, verification, and trace access in a single workflow. It allows engineering teams to achieve live trace visibility in approximately five minutes, making it a rapid solution compared to other tools that require extensive setup and maintenance. The CLI supports Python and TypeScript projects, automatically instruments LLM calls without code changes, and provides a direct path from trace collection to evaluation, integrating cost analysis, online scoring, and CI evaluations into one workflow. Braintrust's free tier permits testing with real production traffic, providing 1 GB of processed data monthly. The tool is particularly advantageous for teams prioritizing setup speed and integration with evaluation workflows, while alternatives like Langfuse and PostHog offer different features that may suit teams with distinct infrastructure needs or existing analytics frameworks.

Braintrust Topics provides a framework for classifying AI agent traces using both built-in and custom facets, which help identify general patterns and domain-specific insights. Built-in facets like Task, Sentiment, and Issues cover basic dimensions of user-agent interactions, but custom facets can offer more granular classifications tailored to specific applications, such as churn risk, citation quality, and tool reliability. Creating effective custom facets involves designing stable label sets, using preprocessors, and coordinating prompts and exclusion patterns to ensure accurate and consistent classifications. The guide emphasizes the importance of refining facets through iterative testing and reviews to address edge cases and improve labeling accuracy. It also underscores the significance of tying each label to a specific workflow to enhance decision-making processes. Custom facets utilize a preprocessor, prompt, and optional exclusion regex to convert traces into labeled clusters, enabling more sophisticated analyses and evaluations of AI interactions.

AI observability is enhanced by Braintrust's new tool, Topics, which is designed to continuously analyze and classify production traces into actionable insights, aiding in the identification of unexpected patterns or failure modes that are difficult to detect manually. By utilizing embedding-based clustering and LLM-assisted labeling, Topics provides dynamic, human-readable classifications that feed into existing evaluation and scoring systems, allowing organizations to act on new findings effectively. These classifications, covering tasks, issues, and sentiment, allow teams to monitor user interactions and system behaviors in real-time, ensuring timely responses to emerging trends or problems. The tool supports custom dimensions, enabling businesses to tailor the analysis to specific needs such as customer segments or compliance categories, and offers integration with CI workflows to automate the process from detection to code deployment. Topics is available across all Braintrust plans, providing a scalable solution for AI teams to discover and address hidden patterns in their data efficiently.

Topics is now generally available, offering a solution to efficiently manage and analyze the vast number of traces generated by customers without incurring high costs or facing scalability issues. The tool employs a simple UI to automatically assign summaries to traces and cluster them, addressing the challenge of processing millions of conversational threads per hour. By collaborating with clients like Replit, the developers of Braintrust have focused on cost-effective intelligence application, enabling active observability and deeper insights through SQL and a user interface. This advancement moves beyond traditional AI observability, as it continuously works in the background to answer questions proactively, without the need for ad-hoc queries. Braintrust aims to adapt to the evolving software development landscape without restricting customers to a specific stack, allowing them to leverage the latest intelligence to transform production data into actionable insights for building high-quality products.

LLM production failures often appear successful in observability tools because they may not trigger exceptions, despite incorrect user-facing answers. To address this, Braintrust provides a system to capture failed traces, label failure modes, and turn them into regression tests for future releases. It ensures that each diagnosed failure, such as hallucinations, retrieval misses, tool-call errors, or format violations, is preserved in a dataset and evaluated using custom scorers both in continuous integration (CI) and on live traffic. This approach highlights the importance of using production traces as the source of truth, allowing engineering teams to convert real-world failures into durable regression tests. By incorporating these traces into regression datasets, Braintrust allows for the detection and prevention of recurring failure patterns, enhancing the reliability and accuracy of LLM systems. The process involves capturing production traces with sufficient context, diagnosing failure modes, promoting traces into datasets, writing appropriate scorers, and integrating them into CI/CD workflows for ongoing evaluation and release gating.