November 2025 Summaries

Retool, an enterprise AppGen platform, has leveraged a data-driven approach to enhance its AI-powered development assistant, Assist, by using production logs as a primary input for roadmap decisions. By employing Braintrust's AI assistant, Loop, Retool queries production data semantically, enabling quick insights and facilitating the prioritization of improvements based on user needs and impact. This shift from an intuitive to a data-informed strategy allows Retool to maintain quality while scaling its user base, as evidenced by the successful prioritization and implementation of features like multi-page support. The team categorizes user intents through a classifier agent, analyzes production data using BTQL, and monitors critical issues with internal dashboards to ensure that engineering efforts align with user demands. Key insights, such as the high demand for multi-page app support and AI-powered README functionalities, were identified through this approach, allowing Retool to address significant user needs efficiently. This transformation underscores the importance of systematic observability in AI development, where production data not only informs debugging but also guides product development, ensuring that resources are directed toward initiatives with the most substantial user impact.

Loop is an AI assistant integrated into Braintrust, designed to enhance evaluation workflows by providing insights from production data, optimizing prompts, and generating custom charts while maintaining accessibility for both technical and non-technical users. Despite promising early metrics, a low acceptance rate for the "Optimize this prompt" feature prompted Mengying from the growth team and design engineer David to investigate user interactions. Through manual reviews, they identified that Loop's default behavior of optimizing all prompts when multiple were present led to user dissatisfaction. To address this, they developed a scoring framework to evaluate user interactions, identified patterns of low satisfaction, and created a targeted dataset for testing solutions. By refining Loop's behavior iteratively and using prompt versioning, they improved user satisfaction, as evidenced by an increase in average conversation scores from 2.1 to 4.3 and reduced computational costs. Their findings highlighted the importance of combining technical and non-technical perspectives in resolving AI product issues and demonstrated how Loop's features can be leveraged to optimize AI applications efficiently.

Loop is an AI assistant integrated into Braintrust, designed to streamline the process of identifying and resolving production issues in AI development. It enables users to ask questions in plain English and provides actionable insights by surfacing patterns and recommendations, generating realistic test cases, and creating custom charts based on production data. Loop simplifies complex queries through semantic searches and offers features like generating BTQL queries, bootstrapping scorers, and compiling datasets from logs for regression testing. It supports cross-functional teams, allowing product managers to analyze production behavior and engineers to debug issues efficiently. Loop's error handling and retry logic ensure a 99.97% success rate in model calls, and its most popular model, gpt-5-nano, accounts for a significant portion of its usage. Loop is widely used across Braintrust's interface, including Logs, Datasets, and Experiments pages, and is particularly effective at querying, analyzing, and taking action on production data.

In the realm of AI observability, traditional metrics of performance such as metrics, logs, and traces have evolved to address the challenges associated with AI systems, which are probabilistic and data-coupled, requiring a focus on traces, evals, and annotation. At Braintrust, a focus on these pillars enables a comprehensive understanding and improvement of AI systems. Traces reconstruct the decision paths across model calls and other components, while evals measure performance both in production and development settings to facilitate systematic improvements. Annotation involves expert input to correct and refine AI behavior, and this data is used to enhance system performance continuously. Braintrust has developed Brainstore, a database purpose-built for handling large-scale AI data, enabling efficient tracing, evaluation, and annotation workflows. This approach shifts the focus from merely determining system uptime to assessing the quality of AI outputs, fostering collaboration among engineers, product managers, and domain experts to ensure that AI systems are both reliable and aligned with user expectations.

Google's release of Gemini 3, a new AI model family, presents advancements in reasoning, tool use, and multimodal capabilities, but its real-world application, especially in agent workflows, requires thorough evaluation beyond standard benchmarks. The process of adopting such models involves establishing a performance baseline with current models using production data, followed by systematic testing of Gemini 3 against real-world scenarios and metrics like tool selection accuracy and response quality. Braintrust facilitates this evaluation by converting production traces into test datasets, allowing for straightforward model comparisons and confident deployment decisions. Continuous monitoring in production ensures that any improvements seen in testing are sustained at scale, with feedback loops integrating performance data to refine future evaluations and deployments. This approach allows AI teams to adapt quickly to new model releases, maintaining a robust cycle of evaluation, deployment, and monitoring to ensure models like Gemini 3 enhance agent performance without introducing regressions.

Prompt evaluation is vital for ensuring that prompts effectively guide language models (LLMs) to produce desired outcomes, as even the most advanced models can falter with poorly designed prompts. As the field evolves, three major trends are shaping prompt evaluation in 2025: the shift from intuition to quantifiable metrics, the mainstream adoption of AI to evaluate AI, and the integration of production as a training ground. Various scenarios, from startups to large enterprises, require tailored evaluation strategies to manage prompt changes, improve AI quality, and maintain compliance. Braintrust emerges as a leading platform by connecting evaluation directly to production monitoring, enabling seamless collaboration between product managers and engineers, and offering tools for prompt experimentation, evaluation, and production monitoring. It stands out with its capability to turn production data into better AI products continuously and measurably, enhancing development velocity and accuracy. Other platforms like LangSmith, Weave, Mirascope, and Promptfoo offer unique features, such as deep integration with LangChain, comprehensive MLOps infrastructure, minimalistic code-centric workflows, and CLI-driven security testing, catering to different team needs and preferences.

Playgrounds in Braintrust facilitate A/B testing by enabling users to compare different prompt variants simultaneously, allowing for the evaluation of prompt improvements through real quality scores before deployment. This process is crucial when iterating on prompts to catch regressions, comparing performance across different models, or evaluating changes against test datasets. The unpredictable effects of prompt development are transformed into measurable comparisons through A/B testing, which is supported both natively in the web interface and via the SDK, catering to both product managers and engineers. Users can experiment visually with prompts, models, and scorers to observe how changes affect key metrics, or they can use code to integrate A/B testing into CI/CD pipelines. By running multiple variants in parallel, Braintrust provides immediate insights into quality scores, latency, token usage, and custom metrics, helping users catch regressions and identify improvements efficiently. The platform supports testing against real-world datasets to ensure representative inputs, and its native CI/CD integration helps prevent regressions from reaching production. Whether using the web UI or the SDK, users can systematically improve prompts and make informed decisions about model performance based on data rather than assumptions.

Retrieval-augmented generation (RAG) systems aim to enhance language model responses by grounding them in relevant documents, but they often encounter challenges such as irrelevant document retrieval, context hallucination, and factually correct but contextually irrelevant answers. Unlike standard LLM evaluation focusing on output quality, RAG evaluation requires assessing the entire pipeline, including retrieval quality, context utilization, and the grounding of answers in source documents. Key evaluation metrics include answer relevancy, faithfulness to retrieved context, context precision, and recall, which measure how well the system retrieves and uses relevant documents to answer questions accurately. Braintrust facilitates RAG evaluation by providing tools for tracing pipeline steps, creating evaluation datasets from real user queries, and employing various scorers to assess different quality dimensions. Continuous evaluation and iteration, including testing retrieval and generation separately and monitoring production performance, are essential for improving RAG systems, as real-world usage reveals edge cases and challenges not apparent in development.

Voice AI agents are transforming customer support, sales, and automated assistance by providing a more natural interface than traditional chatbots, but they introduce complex evaluation challenges like speech quality, conversational flow, and latency. Unlike text-based agents, voice agents must handle background noise, varying accents, and real-time interruptions, requiring comprehensive evaluation across multiple components, such as speech-to-text, natural language understanding, decision logic, response generation, and text-to-speech. Evaluations should measure aspects like speech recognition accuracy, intent classification, response quality, latency, task completion, and user satisfaction, using both offline and online methods to ensure robustness across diverse languages and accents. Continuous monitoring and improvement are crucial to adapting to real-world conditions, as production environments can reveal unexpected issues that offline testing might miss. By systematically refining evaluation datasets and scorers, voice AI agents can be optimized for efficiency, accuracy, and user satisfaction across different contexts and languages.