March 2026 Summaries

In the realm of AI product development, traditional Product Requirements Documents (PRDs) are being replaced by "evals," which are structured, repeatable tests designed to determine if an AI system behaves as intended. Unlike deterministic systems where output is predictable, AI systems produce varied results, necessitating a shift in how product managers define and measure success. Evals act as specs, acceptance criteria, and roadmaps by providing measurable signals for desired outcomes, enabling teams to iterate and improve continuously. The development process involves constructing a "flywheel" where production data feeds back into evals, fostering ongoing enhancement of the AI product. This cycle of observing, analyzing, evaluating, and improving not only accelerates product quality but also creates a durable advantage. Evals employ different types of judges, from algorithmic to AI judges with human alignment, to assess various dimensions of AI output, ensuring that quality improvements align with real-world performance. The role of the AI product manager now includes defining what "good" looks like in code, curating data to highlight deficiencies, maintaining the flywheel for continuous improvement, and safeguarding against regressions. This new approach requires a systematic, data-driven process, where every interaction with the product becomes a potential signal for enhancement, thus ensuring the AI product consistently evolves and improves.

The Braintrust AI Gateway provides a seamless integration of the Claude model into applications using the OpenAI SDK without needing to adopt a separate SDK for Anthropic. This is particularly beneficial for teams already using OpenAI in production who wish to leverage Claude's strengths in tasks like document analysis and multi-step reasoning without the overhead of managing multiple SDKs. Braintrust acts as an intermediary, translating requests from the OpenAI format to Claude's expected format and vice versa, allowing developers to maintain their existing OpenAI integration with minimal changes. By handling the different response formats and error behaviors of OpenAI and Anthropic libraries, Braintrust simplifies the implementation process, enabling developers to switch between AI providers easily by updating model names in the existing setup. Additional features such as logging and caching are supported, enabling performance monitoring and cost tracking for both OpenAI and Claude requests. Braintrust's platform supports various AI providers, allowing developers to manage multiple AI models through a single gateway, ensuring robust and scalable application development.

Braintrust AI Gateway enables developers to use OpenAI models through the Anthropic SDK by converting requests and responses between the two formats, allowing teams to maintain a single SDK integration. This approach simplifies access to multiple language models without additional integration work by handling format conversions and storing provider credentials within Braintrust settings. Engineers can compare outputs from OpenAI's GPT and Anthropic's Claude models using the same client setup, while logging and caching features aid in monitoring performance and cost efficiency. Braintrust supports various SDK and AI provider combinations, including OpenAI, Anthropic, and Google Gemini, along with cloud platform providers, enabling seamless switching between models by adjusting the model name in the existing code. The service ensures that provider credentials remain secure and separate from application code, thus facilitating easy onboarding of new providers and enhancing flexibility in managing AI model integrations.

Braintrust Gateway offers a comprehensive solution for managing and observing large language model (LLM) requests by unifying access to multiple providers such as OpenAI, Anthropic, Google, and AWS, while integrating advanced observability features. It provides a unique capability to capture detailed trace data, including token usage, latency, and costs, without the need for additional instrumentation, facilitating easier debugging and cost analysis. The platform enables developers to use captured traces for evaluation, leveraging features like a playground for testing prompt changes and an AI assistant called Loop to identify quality patterns and cost anomalies. Braintrust also supports integration with OpenTelemetry, allowing seamless incorporation into existing monitoring setups, and offers features like GitHub Actions to prevent regressions by blocking merges that fail quality evaluations. This makes it particularly useful for teams aiming to trace production LLM behavior, debug efficiently, and ensure high-quality deployments, while providing custom cost attribution and the ability to test modifications against real production data directly within the platform.

Self-hosted AI evaluation platforms allow enterprise teams to test, score, and monitor large language model (LLM) outputs within their own infrastructure, ensuring sensitive data remains secure and compliant with regulatory requirements. These platforms, such as Braintrust, Langfuse, Arize Phoenix, and DeepEval, offer various deployment models, including private cloud, on-premises, and hybrid solutions, providing capabilities like trace logging, evaluation workflows, and observability. Braintrust stands out for its hybrid deployment model, which separates the control plane from the data plane, providing enterprise-grade evaluation, compliance, and observability while reducing operational overhead. It supports SOC 2 Type II and HIPAA compliance, integrates with CI/CD platforms, and offers tools for automated and manual evaluations. While open-source options like Langfuse and Arize Phoenix offer more control, Braintrust's managed approach appeals to enterprise teams seeking to streamline infrastructure management and ensure data residency compliance within their VPC, making it a preferred choice for regulated industries.

Traditional observability, which relies on logs, metrics, and traces, is insufficient for AI systems because AI products can appear operationally sound yet produce incorrect or harmful outputs. AI observability addresses this by offering a framework to continuously monitor, evaluate, and enhance AI systems, connecting production behavior tracing, output quality evaluation, and system configuration iteration into a closed loop. The need for AI observability has grown due to the complexity of multi-step AI agents and the inadequacy of traditional testing methods for large-scale AI products. Technical challenges such as managing heterogeneous and mutable trace data, performing exploratory query patterns, and scaling for large data volumes necessitate custom infrastructure, such as Braintrust's Brainstore, which combines multiple database functions into a single system. Effective AI observability platforms tightly integrate various capabilities, including tracing, evaluation, analytics, and automation, providing a seamless feedback loop essential for rapid iteration and improvement.

Braintrust's unified AI Gateway enables developers to access Google's Gemini models using the OpenAI SDK, facilitating seamless integration without modifying existing codebases. By pointing the OpenAI client to Braintrust's gateway URL and using a Braintrust API key, requests are translated into Gemini's API format and returned in the standard OpenAI structure. This allows teams to leverage Gemini's long context windows, multimodal input support, and competitive pricing without the need for a separate SDK or extensive migration work. The gateway also supports logging and caching features, which track request details and store responses for repeated use, enhancing development efficiency. Furthermore, Braintrust's platform supports integration with multiple AI and cloud model providers, enabling teams to standardize on a single SDK and easily switch between models by changing the model name in the request. This centralized approach keeps provider API keys within Braintrust's organization settings, maintaining security and simplifying provider management.

When developing AI features, selecting the right model for a specific use case is critical, as relying on benchmarks alone can yield inconsistent results due to their focus on standardized datasets rather than real inputs. Braintrust facilitates structured model testing by allowing developers to run multiple models on the same inputs and automatically scoring each output, which enables comparisons based on measurable outcomes rather than subjective preferences. The process involves creating a dataset of real user inputs, managing versioned prompts, and comparing models using a unified API that supports various providers. This methodology helps address key factors like output quality, cost, and latency, ensuring that models perform well in real-world scenarios. By organizing results in a test matrix and using built-in scoring tools, Braintrust allows for the detection of performance regressions and supports iterative improvements. The platform also provides a comprehensive UI for managing datasets, prompts, and model comparisons, making it easier for teams to maintain traceability and reproducibility in their AI development workflows.

LLM gateways serve as an intermediary infrastructure layer that simplifies the integration and management of multiple model providers by allowing developers to use a single API endpoint for various models. Notable gateways like Braintrust Gateway stand out by offering a unified API that includes routing, observability, evaluation, and encrypted caching, which are beneficial for developers managing multi-model workflows in one platform. These gateways, such as Braintrust Gateway and OpenRouter, offer features like automatic provider fallbacks, caching, and cost tracking, which reduce the complexity of maintaining separate provider SDKs, authentication setups, and billing systems. Open-source options like LiteLLM and Helicone provide flexibility for self-hosting and integrating additional observability tools, while Portkey caters to enterprise needs with compliance controls and governance features. The gateways differ in deployment options, provider coverage, and pricing models, with some focusing more on routing and model access and others integrating evaluation and observability for a comprehensive development workflow.

Braintrust AI Gateway facilitates seamless integration between Google's Gemini models and Anthropic's SDK, allowing developers to use Gemini models without altering their existing Anthropic SDK setup. By pointing the Anthropic client to the Braintrust gateway URL and authenticating with a Braintrust API key, developers can maintain their usual request patterns while accessing Gemini models, as Braintrust handles the format conversion and response structure. This integration eliminates the need to support multiple SDKs and request logic, enabling teams to switch models by simply changing the model name in the existing code. Braintrust supports a variety of AI providers, including OpenAI, Anthropic, Google, and more, and offers a centralized place to manage provider credentials, which are securely stored and kept out of application code. Additionally, Braintrust provides logging and caching features to track requests, token usage, and costs, thereby simplifying the management of multiple models and providers within a single application framework.

AI evaluations (evals) serve as a structured method to measure the quality of AI features, addressing the challenge of non-deterministic outputs that cannot be manually tested. For product managers, evals provide coverage by testing a vast array of scenarios, tradeoff visibility to understand how improvements in one area may affect another, and version comparison to confirm enhancements over previous iterations. The process involves three main components: datasets that represent real user interactions, tasks that are evaluated, and scorers that measure quality across specific dimensions. These evaluations are crucial for making informed product decisions by providing data-backed insights rather than relying solely on engineering assessments. Evals require collaboration among product managers, AI engineers, subject matter experts, and data analysts, and they can be integrated into a continuous improvement loop to enhance AI systems effectively. Tools like Braintrust support this process by allowing various roles to collaborate seamlessly and leverage AI assistants like Loop to accelerate workflows, identify patterns, and optimize tasks without coding, ultimately creating a sustainable evaluation workflow.

Starteris introduces a new usage-based scaling plan designed for growing teams, allowing them to continue building without a platform fee by paying only for data usage. It complements existing Free and Pro plans, with Pro offering advanced features like environments, custom charts, and more predictable workflows for a monthly fee. The plan changes do not affect current users unless they choose to switch, and transitions between plans are straightforward through the billing settings. For teams requiring enterprise-level features such as SSO and compliance certifications, an Enterprise plan is available upon contact. Existing Free and Pro users maintain their current pricing and features unless they opt for a different plan.

Braintrust offers a platform designed to meet the complex data security and compliance needs of organizations, particularly those subject to the EU's General Data Protection Regulation (GDPR). By architecturally separating the control plane from the data plane, Braintrust provides flexibility in data storage and governance, allowing customers to host their data in environments that satisfy local regulatory requirements and their own security policies. This design enables GDPR compliance by allowing customer data to remain within chosen jurisdictions or environments, while the control plane remains a managed service offering the benefits of SaaS such as updates and a unified UI. This separation is particularly beneficial for EU teams and global organizations operating under diverse jurisdictional data residency requirements, as it combines the speed and convenience of SaaS with the security and governance advantages of self-hosted infrastructure.

Grafana and Braintrust are two platforms that together provide a comprehensive observability and evaluation framework for large language model (LLM) applications. Grafana is the industry standard for monitoring the health of LLM infrastructure, offering visibility into metrics such as latency, error rates, token usage, and GPU utilization, while also conducting basic safety evaluations through OpenLIT. Braintrust complements Grafana by filling the evaluation gap, offering tools to score output quality, manage prompt versions, run regression tests, and enforce CI/CD quality gates. It can identify issues like prompt regressions quickly and links production traces to scoring workflows, making it the stronger choice for ensuring that LLM outputs meet business and policy standards. Through OpenTelemetry, the two platforms can be integrated to provide a full-stack monitoring and evaluation system, with Grafana focusing on infrastructure health and Braintrust ensuring model output quality. Companies like Notion and Stripe use Braintrust to improve their LLM evaluation and observability capabilities, benefiting from features such as custom scoring functions, exhaustive trace logging, and prompt version management.

Enterprise teams often rely on Datadog for LLM visibility due to its robust monitoring capabilities, which include tracing, operational monitoring, and managed evaluations. However, Datadog's features are primarily focused on observability rather than structured evaluation, lacking integration with CI pipelines and release control processes necessary for ensuring LLM output quality before deployment. Braintrust is designed to fill this gap by offering CI-integrated evaluation workflows and comprehensive evaluation governance, enabling teams to conduct prompt version management, regression enforcement, and production-to-dataset workflows within a unified platform. Braintrust's approach allows for real-time evaluation and quality control, embedding these processes into CI pipelines and release decisions, thus providing a more cohesive solution for teams prioritizing LLM output quality and reliability. While Datadog remains a strong choice for infrastructure monitoring, Braintrust offers a more integrated solution for managing the full lifecycle of LLM evaluation and release, making it a preferred option for organizations treating LLM quality as a critical deployment requirement.

Various prompt playgrounds and management tools are tailored for product managers (PMs) to iterate effectively on prompts, models, parameters, and tools without requiring engineering intervention. Braintrust is highlighted for its no-code workspace that integrates evaluation workflows directly, enabling real-time experimentation and side-by-side comparison of prompt variants. Vellum focuses on visual prompt engineering with seamless deployment, while Arize provides the ability to test prompts with real production data, offering a higher level of confidence for PMs. Langfuse caters to teams needing open-source flexibility, whereas Humanloop emphasizes cross-functional collaboration with governance and CI/CD integration. PromptLayer offers strong capabilities for no-code prompt editing and debugging, and Promptfoo targets developer-first environments with a CLI-first approach for rigorous evaluations. Each tool offers unique features such as version control, replay workflows, and side-by-side diff comparisons, allowing PMs to efficiently manage prompt iterations and ensure performance improvements before deployment.

Braintrust enhances AI observability by going beyond basic logging tools like Grafana and Datadog, which primarily focus on operational metrics such as latency and token usage. Unlike traditional logging, which merely confirms the completion of requests, Braintrust evaluates responses against quality standards, identifying failures in the execution chain and providing a structured path for resolving production issues. This approach allows developers to detect and verify improvements before these reach end-users, ensuring higher accuracy, relevance, and safety of AI outputs. Braintrust integrates evaluation directly into production workflows, offering automated scoring, prompt versioning, and CI quality gates to maintain high standards and prevent regressions. It maps every execution step, from retrieval to final output, enabling detailed root-cause analysis and prompt lifecycle management, which ensures that quality improvements are consistently enforced. This comprehensive governance layer allows organizations like Notion and Stripe to manage AI systems effectively, using Braintrust to align evaluation results with deployment decisions, thereby ensuring output correctness in line with business requirements.

In this detailed guide, the author shares a systematic approach to developing an evaluation (eval) process for improving AI systems, specifically focusing on generating quality Mermaid diagrams. The guide outlines a ten-step method to transform an initial "vibe" or feeling about system performance into a measurable and improvable evaluation framework. Key steps include defining the system and the initial feeling, forming clear questions, building a small dataset, defining and implementing scorers, iterating on prompts, and using AI tools to speed up the iteration process. The author emphasizes the importance of understanding system constraints, balancing multiple evaluation dimensions to avoid overfitting, and managing variance and nondeterminism in outputs. By following this process, the author was able to enhance the quality of Mermaid diagrams generated by their AI model, establishing a reliable system for continuous improvement and evaluation.

Braintrust emerges as a comprehensive platform for measuring AI agents in production, distinguishing itself from Promptfoo, which is primarily designed for local, CLI-driven prompt testing and red-teaming. While Promptfoo is suitable for individual developers due to its local evaluation capabilities, it lacks the features necessary for team collaboration, production monitoring, and persistent experiment history, which Braintrust provides. Braintrust integrates seamlessly with various frameworks and SDKs, offers AI-assisted evaluation through Loop, and ensures quality through CI/CD gates, making it ideal for teams that need to track and improve their AI agents' performance in real-time production environments. Braintrust also supports a broader evaluation lifecycle, from production data to deployment gates, addressing needs beyond Promptfoo's capabilities. For those needing specific metrics and scoring, alternatives such as DeepEval and RAGAS offer targeted evaluation for Python-native and RAG pipelines, respectively.

Braintrust is positioned as a leading alternative to LangSmith for production teams that require comprehensive evaluation-to-production workflows with integrated CI/CD quality gates, structured evaluations, and controlled release mechanisms. Unlike LangSmith, which is tightly integrated with LangChain, Braintrust offers broader framework compatibility, allowing teams to utilize their existing infrastructure without needing to adopt specific orchestration frameworks. It converts production failures into regression tests and ensures evaluation results directly influence release decisions, thus preventing regressions from reaching users. While LangSmith provides visibility into model performance, it relies on manual review to act on evaluation results. In contrast, Braintrust enforces quality standards automatically before deployment, facilitating collaboration between product and engineering teams within the same environment and offering a shared playground for testing prompt changes. This approach is favored by companies such as Notion and Stripe, which have improved their issue identification and resolution capabilities by employing Braintrust's workflow. Braintrust’s model supports unlimited collaborators in its free tier, providing cost-effective scaling for teams focused on enhancing evaluation practices.

Braintrust emerges as a leading alternative to DeepEval, offering comprehensive coverage of the evaluation lifecycle, including production monitoring, team collaboration, and automated release enforcement within a single platform. While DeepEval is effective for local testing and provides a variety of built-in metrics for evaluating large language models (LLMs), it lacks the infrastructure for production monitoring and shared dashboards, which Braintrust addresses. Other alternatives like RAGAS, Promptfoo, LangSmith, Langfuse, Vellum, and Galileo each offer niche capabilities such as research-backed metrics, red teaming, LangChain integration, self-hosting, visual workflow design, and real-time guardrails, respectively. However, they do not provide the unified governance layer that Braintrust offers, which directly connects evaluation outcomes to deployment decisions, ensuring quality standards are maintained across the development and production phases. Braintrust's ability to capture production traces, convert failure cases into structured datasets, and integrate scoring into CI/CD processes makes it particularly appealing for organizations that prioritize maintaining consistent quality and preventing regressions in their deployments.

Braintrust is highlighted as an effective debugging platform for AI agents in production, integrating trace inspection, evaluation, and CI/CD enforcement into a unified workflow. The platform facilitates the transformation of production failures into permanent evaluation cases, preventing future regressions by validating every code change. Braintrust supports over 40 framework integrations and provides a native GitHub Action for automated evaluations, making it suitable for diverse tech stacks. The guide contrasts Braintrust with other tools like LangSmith, which is more focused on LangChain and LangGraph ecosystems, and explains key differences between debugging, monitoring, and observability. Debugging AI agents involves tracing, isolating, and resolving errors in multi-step workflows, emphasizing the need for reconstructing execution paths to identify and fix issues. The document also discusses how effective debugging prevents the recurrence of failures by incorporating them into automated evaluation suites, thereby enhancing the reliability of agent deployments.