July 2026 Summaries

Behavior specs, introduced by Braintrust and Basis, provide an open standard for defining and evaluating the behavior of AI agents, particularly those that operate over long trajectories. These specs aim to shift the focus from merely assessing outcomes to supervising the processes agents undergo to reach those outcomes, thus ensuring more reliable and trustworthy AI performance. Unlike traditional outcome evaluations that can be expensive and fail to capture the nuances of complex decision-making, behavior specs allow for a detailed examination of each step in an agent's trajectory, identifying potential errors and overfitting. This approach is akin to process supervision in human organizations, where following established procedures is believed to produce better results. For AI agents, behavior specs make expectations explicit, guiding agents on how they should operate in specific situations. They are designed to be continuously tested and updated, ensuring alignment with intended behaviors and preventing drift over time. By focusing on process rather than just outcome, behavior specs help teams build AI agents that are both efficient and effective, with the flexibility to adapt as models improve. The open-source nature of these specs, accessible at agentbehavior.dev, encourages widespread adoption and customization, enabling organizations to establish their own standards for AI behavior.

AI coding tools can be categorized into three primary types: standalone AI editors, extensions for existing IDEs, and command-line or delegated agents, each offering different levels of integration and functionality in the software development process. Standalone editors like Cursor and Devin Desktop replace traditional coding environments, providing features such as inline suggestions, multi-file edits, and project-aware chat, while extensions like GitHub Copilot enhance existing IDEs with AI-driven completions and chat features without altering the current workflow. Command-line agents such as Claude Code and Codex operate from the terminal, performing multi-step tasks across repositories with capabilities for editing, testing, and reviewing code through diffs and logs. The choice among these tools often depends on developers' preferences for workflow integration, the extent of repository access required, and the timing of human review. Each tool offers different pricing models, with free plans providing limited access and paid plans offering expanded usage and capabilities, allowing teams to select the best fit based on their development needs and existing infrastructure.

No-code AI agent builders offer a visual interface for creating AI workflows, reducing the need for extensive engineering work by replacing orchestration code with visual canvases, plain-language configurations, and prebuilt templates. These builders, which include platforms like Lindy, Relevance AI, Stack AI, Gumloop, and Bardeen, enable users to define tasks, connect tools such as Gmail and Slack, and configure triggers, thereby managing the execution sequence and conditional logic of workflows. While they provide rapid deployment, the complexity and stakes of the tasks dictate the required level of control, with higher-stakes tasks benefiting from code-based frameworks due to enhanced control over execution, debugging, and failure handling. Each builder targets specific use cases, from personal assistants and enterprise deployments to structured automations and web research, allowing teams to choose based on their specific workflow needs and the level of technical control required. As tasks become more complex or business-critical, teams may transition to code-based solutions for greater reliability, control, and evaluation infrastructure, with platforms like Braintrust offering observability and evaluation layers to ensure quality and compliance in production environments.

Building AI agents involves creating systems that use language models to perform tasks through multiple steps, with the ability to decide actions, call tools, and carry context between operations until reaching a desired outcome. Developers face challenges such as managing the model loop, state, memory, and error handling, but agent frameworks provide components that streamline these processes, allowing teams to concentrate on task logic and data access. The text outlines various frameworks for building AI agents, each offering unique features suited to different needs: LangGraph for complex, state-controlled agents; CrewAI for role-based, multi-agent teams; OpenAI Agents SDK for lightweight, OpenAI-focused applications; Pydantic AI for type-safe Python services; and Mastra for TypeScript-native workflows. These frameworks integrate with Braintrust for testing and monitoring, enabling detailed tracing and evaluation of agent performance. The choice of framework depends on the application's execution pattern, language compatibility, and the level of control required over agent operations.

Standard logging practices for Large Language Models (LLMs) often fall short during audits due to incomplete records, short retention periods, editable logs, difficult export processes, and broad access, which compromise the ability to provide proof of specific interactions. Audit-ready LLM logs must capture comprehensive details such as the full prompt and response, tool activity, model versions, timestamps, user identifiers, metadata, and ensure records are tamper-evident. Braintrust offers solutions by structuring these logs into spans that include tool calls, responses, and metadata for easier inspection and compliance. It supports custom retention policies, immutability features, and export paths to streamline audit and incident review processes. Access control is managed through role-based permissions and hybrid deployments, enabling organizations to maintain data within their own infrastructure, which is crucial for adhering to regulations like HIPAA, SOC 2, and GDPR. The platform provides flexibility in deployment and retention settings to meet varied compliance needs while ensuring data security and audit readiness.

Traditional logging methods fall short for Large Language Model (LLM) applications because they are designed for deterministic systems, where a single log line can reveal sufficient information about an event or outcome. In contrast, LLM applications involve complex and non-deterministic processes, often requiring multiple steps such as retrieval, tool calls, and model interactions, which makes it difficult to trace errors or unexpected outputs through logs alone. LLM tracing provides a complete, structured record of a request by breaking it into interconnected spans, each representing a distinct operation, thus offering a clearer picture of the execution path and the ability to identify where errors or incomplete context occurred. This tracing approach is essential for debugging and optimizing LLM applications, as it captures detailed information about each step, including inputs, outputs, timing, and token usage, which is crucial for understanding the "why" behind a response, not just the "what." Braintrust offers a comprehensive solution for LLM tracing, enabling teams to incorporate tracing into their workflows seamlessly, leveraging both SDK and OpenTelemetry integrations, and providing tools for online scoring, monitoring, and regression testing to ensure quality and efficiency in LLM applications.

In 2026, the choice of an AI agent framework is critical, guided by the specific workflow needs of the application, and frameworks must align with the agent's control flow, runtime, and production requirements before considering secondary features. LangGraph excels with stateful, multi-step agents requiring explicit control, while CrewAI supports role-based collaboration, and the OpenAI Agents SDK is designed for linear handoff chains. LlamaIndex is ideal for retrieval-heavy agents, and Mastra suits TypeScript applications with model-driven agent loops. Python remains a strong choice for research and niche model integrations, whereas TypeScript is preferable for applications running on JavaScript runtimes. Ecosystem maturity, community adoption, and stable releases are essential for reducing implementation risk. Braintrust emerges as a preferred evaluation layer, capable of integrating with multiple frameworks like LlamaIndex, Mastra, LangGraph, CrewAI, and OpenAI Agents SDK, ensuring consistent evaluation across mixed framework stacks. This integration enables the capture of distinct trace structures, aiding in maintaining quality and reliability as orchestration choices evolve.

In a comparison between the Paper and Figma MCP servers for frontend agents, Paper demonstrates more consistent performance across different page complexities, while Figma's performance varies significantly, particularly on complex designs. Paper maintains quality from simple to complex pages, whereas Figma shows a decline, although both tools tie on visual similarity in many cases. The evaluation reveals that Figma's semantic HTML output makes it more accessible and maintainable compared to Paper's div-heavy approach. However, Paper proves to be faster and more cost-effective, with 42% shorter run times and 32% lower costs per point of visual quality. Figma experiences occasional major failures on certain design types, such as 3D character art and flat illustrations, contributing to its inconsistency. Despite these differences, both tools render pages similarly well, and self-checking behaviors observed in agents do not significantly improve outcomes. The evaluation highlights the inherent limitations of CSS in replicating complex visual elements typically created in 3D rendering software.

Logs are insufficient for diagnosing root causes of failures in large language model (LLM) systems because they capture only the final output without detailing the intermediate steps necessary for understanding where the error occurred. Tracing, as opposed to logging, involves recording the entire execution path—including retrieval, prompt assembly, model calls, and tool arguments—allowing engineers to pinpoint the exact step where the failure originated. Braintrust addresses this by capturing detailed trace trees for each AI call, including metadata about input prompts, model outputs, and configuration details, which facilitates root cause analysis. The process involves reproducing the failure, navigating through the trace tree to identify the failing span, isolating the problematic layer, and confirming the fix with regression tests. This method not only aids in understanding why an incident occurred but also ensures that similar issues are prevented in future deployments by integrating the failing trace into a regression dataset, thereby enhancing the reliability of LLM systems.

Topics utilizes a small, cost-effective model to achieve active observability by reading and summarizing production traces with a large language model (LLM) and clustering the summaries, allowing users to monitor what their agents are doing without manually reviewing logs. The development and optimization of this model involved collaboration between Baseten and Braintrust, focusing on balancing affordability and quality, with iterations primarily targeting the summarization step. An off-the-shelf Gemma 4B model was initially used, and through prompt adjustments and benchmarking, the model was refined to improve label correctness and recall of issues at a fraction of the cost of frontier models. The production setup achieved 82.2% label correctness and demonstrated that a well-crafted small model, paired with accurate prompts and examples, can outperform more expensive models on specific metrics. This approach to model optimization and active observability is not unique to Topics and can be applied to other products that rely on LLMs and large datasets.

A vector database is integral to retrieval-augmented generation (RAG) applications by storing and managing embeddings derived from source documents, which are then queried in response to user inputs to provide relevant context for language models. These databases are categorized into managed, self-hosted, or hybrid models, exemplified by options such as Pinecone, Weaviate, Qdrant, Chroma, and Turbopuffer, each offering unique strengths and trade-offs regarding scalability, operational control, and specific application needs. Managed services like Pinecone reduce infrastructure overhead but may incur higher costs with increased usage, while open-source options like Weaviate and Qdrant provide more control at the expense of additional operational responsibilities. Chroma offers an accessible starting point for smaller applications and prototypes, whereas Turbopuffer emphasizes cost-effective storage for large datasets. The choice of vector database depends largely on the team's preference for operational simplicity versus control, the scale of data, and specific retrieval and search requirements, with tools like Braintrust available to assess the quality and effectiveness of retrieval in the RAG pipeline.

AI customer service agents are sophisticated tools that streamline customer interactions by understanding requests, retrieving relevant business data, and either resolving issues or escalating them to human agents when necessary. These agents can operate autonomously across multiple communication channels such as chat, email, voice, and social media, and are integrated with existing business systems to ensure seamless operation. The platforms vary in their approaches, with some like Sierra and Decagon offering standalone solutions that connect to various business systems, while others like Intercom Fin and Zendesk AI agents work within existing help desk environments to enhance workflow efficiency. Ada provides a broad customer experience automation platform capable of handling multilingual support across diverse channels. These platforms are designed to cater to different organizational needs, from high-volume support teams requiring detailed analytics and quality monitoring, to businesses seeking to maintain brand voice across customer interactions. The choice of platform depends on factors such as the existing support infrastructure, channel requirements, and the degree of autonomy desired, with ongoing evaluation crucial to ensure accuracy and compliance with policies.

An AI API acts as an intermediary between applications and hosted model runtimes, facilitating tasks like provider-side inference and managing operational metadata without requiring teams to handle infrastructure like GPUs or deployment scaling. Notable providers such as Groq, Cerebras, Fireworks AI, Together AI, and Baseten offer diverse features tailored to different needs, including low-latency performance, high token throughput, broad model catalogs, and deployment control. Each provider charges primarily per token with varying price points depending on factors like model size and request type, and they offer different options for fine-tuning and dedicated infrastructure. These APIs are suitable for a range of applications, from real-time chat and voice agents to high-volume generation and custom model deployments, with the Braintrust Gateway offering a unified way to log and compare provider performance. Pricing, speed, and suitability for specific use cases depend on factors such as the model, region, and traffic volume, and users are encouraged to test providers with their own workloads to make informed decisions.

AI agent reliability is essential for ensuring that AI systems complete tasks accurately across all workflow steps, with a focus on preventing compounded errors in long processes. Reliability is assessed through a loop involving pre-deployment evaluations, production observability, and regression debugging, where each stage informs the next to maintain consistency and improve performance. Braintrust is highlighted as a leading AI agent reliability tool, offering an integrated system for pre-deploy evaluations, production traces, online scoring, and regression debugging, with a focus on using the same scorer throughout the development and production phases. The tool's capabilities enable teams to transform production failures into regression tests, ensuring that agent quality is consistently measured and enforced. Braintrust's approach is contrasted with other tools like Galileo, Arize Phoenix, Promptfoo, and AgentOps, which serve different needs such as runtime guardrails, open-source tracing, and session replay. The importance of having a comprehensive reliability tool that supports various frameworks and integrations is emphasized to prevent production failures from recurring, with Braintrust also offering a free tier for teams to start enhancing AI reliability.

A fine-tuning platform for large language models (LLMs) allows teams to specialize a general open model for specific tasks by continuing training on application-specific data, which reduces reliance on prompt instructions. The platforms facilitate creating a stable model that consistently follows desired behaviors, like a support classifier maintaining label consistency or a data extraction model adhering to a JSON schema. Fine-tuning options include LoRA and QLoRA, which are cost-efficient but require less control, and full fine-tuning, which offers more control at the cost of increased computing resources. Managed fine-tuning platforms handle infrastructure needs, while self-hosted frameworks offer control over resources and data. Among the fine-tuning platforms discussed are OpenPipe, which is noted for converting application data into tuned models to reduce costs; Predibase, which efficiently serves multiple adapters; Together AI, which integrates fine-tuning and inference; Axolotl, which offers full control over the training environment; and Baseten, focusing on deployment and serving. The choice between managed and self-hosted solutions depends on the team's priorities regarding infrastructure control, cost, and operational requirements.

The GPT-5.6 family, comprising Sol, Terra, and Luna models, has been evaluated alongside Anthropic's Fable, Opus 4.8, and Sonnet 5 against 225 machine-checkable tasks across arithmetic, symbolic rules, and data transformation categories. Sol is identified as the most consistent performer, particularly in symbolic rules, while Terra offers similar quality with reduced latency, making it ideal for latency-sensitive tasks. Luna excels in data transformation tasks but struggles with symbolic rules. The evaluation highlights cost-effectiveness and solve rates, with Sol leading in accuracy but Terra and Luna providing competitive alternatives for decomposed subtasks. Anthropic's models show lower scores primarily due to refusal rates rather than incorrect answers, with Fable demonstrating high accuracy when tasks are attempted. The study emphasizes the need for tailored model selection based on task complexity and specific operational requirements, suggesting Sol for complex planning and Terra or Luna for executing simpler, decomposed tasks.

In an in-depth evaluation of six speech-to-text (STT) models conducted across 240 audio cases and eight content domains, the study highlights the importance of selecting the right STT provider for voice agents by focusing on transcription accuracy and its impact on downstream responses. The evaluation used a comprehensive methodology that included assessing transcription similarity, critical entity recall, and answer equivalence, alongside real-time latency measurements. The study found that while all models were closely matched in accuracy, OpenAI's gpt-4o-transcribe emerged as the top choice, offering the best balance of answer quality and low latency. It emphasized the value of using domain-specific vocabularies and post-transcription corrections to enhance the performance of STT systems, particularly for structured tokens like IDs and callsigns, which prove challenging for many models. Additionally, the research underscored the need to incorporate audio review in the evaluation process to differentiate between genuine errors and reference issues, suggesting that the choice of model should align with project-specific priorities, whether it be accuracy, speed, or the preservation of critical information.

Brainstore, a database tailored for handling agent traces, faced challenges with traditional phrase search in large datasets, which often resulted in slow queries due to common terms with rare intersections. The solution involved implementing shingled bloom filters using trigrams instead of unigrams, improving segment elimination by focusing on rare three-word combinations rather than individual common words. This approach significantly enhanced search efficiency, allowing for faster phrase search by pruning irrelevant data more effectively. The improved method was tested on real customer data, reducing the scanned data size from over 100 GB to less than 4 GB, resulting in a 25x increase in efficiency. As Brainstore continues to develop, it aims to further optimize its search capabilities and handle larger datasets, ensuring that agent debugging remains fast and efficient even as data volumes grow.

In preparation for the USA vs Belgium Round of 16 match in the 2026 FIFA World Cup, a detailed evaluation was conducted using the monolithic-pro configuration of a research tool, which was previously identified as cost-effective and efficient for football match analysis. This setup, which provides comprehensive mapping at a lower cost compared to other configurations, focuses on key elements like squad structure, player availability, recent form, and historical team relationships. Notably, the research highlights Belgium as the favorite with a 68% predicted chance of victory, emphasizing the tactical roles of key players such as De Bruyne and Lukaku, despite some injury concerns. The tool's analysis shows that the U.S. team faces challenges due to potential player availability issues and historical disadvantages. The research underscores the importance of inspecting and updating underlying evidence to ensure accuracy, as demonstrated by the update on Folarin Balogun's red-card status, which was lifted prior to the match. The study ultimately offers a clearer pre-match picture, leveraging insights from previous evaluations to enhance understanding and prediction accuracy.

LLM observability in TypeScript requires a comprehensive tracing setup that provides a TypeScript SDK, supports various runtimes, and offers detailed insights into request paths and failures. Effective tracing should capture inputs, outputs, latency, and errors at each step, allowing teams to isolate issues in model calls, tool invocations, and runtime operations. Auto-instrumentation can simplify the integration of tracing into TypeScript applications, while manual instrumentation offers more control. The Vercel AI SDK, enhanced by Braintrust, enables detailed tracing and telemetry, facilitating debugging and evaluation of production traces. This setup supports not only model and tool call tracing but also the transformation of production traces into reusable evaluation datasets. By capturing comprehensive request data, teams can utilize these traces for testing and improving future releases, ensuring that any production failures identified are addressed before they affect users again. For optimal results, teams should select tracing tools that align with their app's runtime, framework, and quality assurance processes.

The text discusses the importance and methodology of tracing in Python applications, particularly those involving large language models (LLMs), to ensure smooth and efficient operation. It emphasizes the need for a comprehensive trace that captures various steps such as retrieval, preprocessing, model calls, and more, enabling teams to identify and resolve issues efficiently. The use of OpenTelemetry is highlighted as a foundational tool that provides a standardized approach for creating and transporting spans, but with additional LLM-specific interpretation required for effective debugging and evaluation. Braintrust is presented as a robust solution for Python-native instrumentation, allowing teams to trace LLM applications by automatically capturing inputs, outputs, latency, and costs, while also integrating with existing OpenTelemetry setups. It supports both auto and manual instrumentation, offering flexibility in tracing provider and framework calls as well as application-specific logic. Additionally, the text explains how traces can be transformed into evaluation datasets to enhance future release checks, using real production cases to ensure continued application reliability.

Traditional Application Performance Monitoring (APM) tools like Datadog, Grafana, and Honeycomb often fall short in effectively monitoring Large Language Model (LLM) applications, as they focus primarily on metrics such as latency, error rates, and system health, which may not reflect the quality of the model's output. OpenTelemetry offers a solution by integrating structured telemetry at the LLM layer, capturing detailed data about prompts, retrievals, tool calls, and model responses, which standard APM tools typically overlook. By implementing OpenTelemetry's GenAI semantic conventions, organizations can trace and evaluate LLM applications more effectively, ensuring output quality and system observability are interconnected. This approach allows teams to route spans to multiple backends, such as Braintrust for output scoring and traditional APM tools for operational monitoring, without needing to modify existing telemetry paths. Through distributed tracing and the use of both automatic and manual spans, OpenTelemetry provides a comprehensive view of LLM workflows, enabling teams to debug, assess quality, and turn production failures into test cases, thereby maintaining robust and reliable LLM application pipelines.

During the 2026 World Cup, the Braintrust team explored the effectiveness of Parallel's web research agents in automating the data collection and analysis typically done by analysts and fans when examining football matchups. They utilized Parallel Web Systems’ Task API to create structured, source-backed maps of squad dynamics, including player availability, head-to-head records, and recent performance, which were then organized into knowledge graphs for easy inspection. The research experiments involved running 48 matchups through six different configurations, using two architectures—monolithic and fan-out—across three processing tiers to assess how research depth and task design interact. The findings suggested that while the monolithic-pro configuration offered a cost-effective solution with substantial coverage, the fan-out approach excelled in domain-specific tasks like injury analysis. Despite improvements in research depth, prediction calibration remained consistent across configurations, indicating that while the process provided a detailed and inspectable overview, it did not necessarily enhance forecast accuracy. The study demonstrated the utility of graph-based structures in making football intelligence evaluable, allowing for ongoing application throughout the tournament as team compositions and conditions evolved.