July 2026 Summaries

Topics utilizes a small, cost-effective model to achieve active observability by reading and summarizing production traces with a large language model (LLM) and clustering the summaries, allowing users to monitor what their agents are doing without manually reviewing logs. The development and optimization of this model involved collaboration between Baseten and Braintrust, focusing on balancing affordability and quality, with iterations primarily targeting the summarization step. An off-the-shelf Gemma 4B model was initially used, and through prompt adjustments and benchmarking, the model was refined to improve label correctness and recall of issues at a fraction of the cost of frontier models. The production setup achieved 82.2% label correctness and demonstrated that a well-crafted small model, paired with accurate prompts and examples, can outperform more expensive models on specific metrics. This approach to model optimization and active observability is not unique to Topics and can be applied to other products that rely on LLMs and large datasets.

AI customer service agents are sophisticated tools that streamline customer interactions by understanding requests, retrieving relevant business data, and either resolving issues or escalating them to human agents when necessary. These agents can operate autonomously across multiple communication channels such as chat, email, voice, and social media, and are integrated with existing business systems to ensure seamless operation. The platforms vary in their approaches, with some like Sierra and Decagon offering standalone solutions that connect to various business systems, while others like Intercom Fin and Zendesk AI agents work within existing help desk environments to enhance workflow efficiency. Ada provides a broad customer experience automation platform capable of handling multilingual support across diverse channels. These platforms are designed to cater to different organizational needs, from high-volume support teams requiring detailed analytics and quality monitoring, to businesses seeking to maintain brand voice across customer interactions. The choice of platform depends on factors such as the existing support infrastructure, channel requirements, and the degree of autonomy desired, with ongoing evaluation crucial to ensure accuracy and compliance with policies.

An AI API acts as an intermediary between applications and hosted model runtimes, facilitating tasks like provider-side inference and managing operational metadata without requiring teams to handle infrastructure like GPUs or deployment scaling. Notable providers such as Groq, Cerebras, Fireworks AI, Together AI, and Baseten offer diverse features tailored to different needs, including low-latency performance, high token throughput, broad model catalogs, and deployment control. Each provider charges primarily per token with varying price points depending on factors like model size and request type, and they offer different options for fine-tuning and dedicated infrastructure. These APIs are suitable for a range of applications, from real-time chat and voice agents to high-volume generation and custom model deployments, with the Braintrust Gateway offering a unified way to log and compare provider performance. Pricing, speed, and suitability for specific use cases depend on factors such as the model, region, and traffic volume, and users are encouraged to test providers with their own workloads to make informed decisions.

AI agent reliability is essential for ensuring that AI systems complete tasks accurately across all workflow steps, with a focus on preventing compounded errors in long processes. Reliability is assessed through a loop involving pre-deployment evaluations, production observability, and regression debugging, where each stage informs the next to maintain consistency and improve performance. Braintrust is highlighted as a leading AI agent reliability tool, offering an integrated system for pre-deploy evaluations, production traces, online scoring, and regression debugging, with a focus on using the same scorer throughout the development and production phases. The tool's capabilities enable teams to transform production failures into regression tests, ensuring that agent quality is consistently measured and enforced. Braintrust's approach is contrasted with other tools like Galileo, Arize Phoenix, Promptfoo, and AgentOps, which serve different needs such as runtime guardrails, open-source tracing, and session replay. The importance of having a comprehensive reliability tool that supports various frameworks and integrations is emphasized to prevent production failures from recurring, with Braintrust also offering a free tier for teams to start enhancing AI reliability.

A vector database is integral to retrieval-augmented generation (RAG) applications by storing and managing embeddings derived from source documents, which are then queried in response to user inputs to provide relevant context for language models. These databases are categorized into managed, self-hosted, or hybrid models, exemplified by options such as Pinecone, Weaviate, Qdrant, Chroma, and Turbopuffer, each offering unique strengths and trade-offs regarding scalability, operational control, and specific application needs. Managed services like Pinecone reduce infrastructure overhead but may incur higher costs with increased usage, while open-source options like Weaviate and Qdrant provide more control at the expense of additional operational responsibilities. Chroma offers an accessible starting point for smaller applications and prototypes, whereas Turbopuffer emphasizes cost-effective storage for large datasets. The choice of vector database depends largely on the team's preference for operational simplicity versus control, the scale of data, and specific retrieval and search requirements, with tools like Braintrust available to assess the quality and effectiveness of retrieval in the RAG pipeline.

A fine-tuning platform for large language models (LLMs) allows teams to specialize a general open model for specific tasks by continuing training on application-specific data, which reduces reliance on prompt instructions. The platforms facilitate creating a stable model that consistently follows desired behaviors, like a support classifier maintaining label consistency or a data extraction model adhering to a JSON schema. Fine-tuning options include LoRA and QLoRA, which are cost-efficient but require less control, and full fine-tuning, which offers more control at the cost of increased computing resources. Managed fine-tuning platforms handle infrastructure needs, while self-hosted frameworks offer control over resources and data. Among the fine-tuning platforms discussed are OpenPipe, which is noted for converting application data into tuned models to reduce costs; Predibase, which efficiently serves multiple adapters; Together AI, which integrates fine-tuning and inference; Axolotl, which offers full control over the training environment; and Baseten, focusing on deployment and serving. The choice between managed and self-hosted solutions depends on the team's priorities regarding infrastructure control, cost, and operational requirements.

The GPT-5.6 family, comprising Sol, Terra, and Luna models, has been evaluated alongside Anthropic's Fable, Opus 4.8, and Sonnet 5 against 225 machine-checkable tasks across arithmetic, symbolic rules, and data transformation categories. Sol is identified as the most consistent performer, particularly in symbolic rules, while Terra offers similar quality with reduced latency, making it ideal for latency-sensitive tasks. Luna excels in data transformation tasks but struggles with symbolic rules. The evaluation highlights cost-effectiveness and solve rates, with Sol leading in accuracy but Terra and Luna providing competitive alternatives for decomposed subtasks. Anthropic's models show lower scores primarily due to refusal rates rather than incorrect answers, with Fable demonstrating high accuracy when tasks are attempted. The study emphasizes the need for tailored model selection based on task complexity and specific operational requirements, suggesting Sol for complex planning and Terra or Luna for executing simpler, decomposed tasks.

In an in-depth evaluation of six speech-to-text (STT) models conducted across 240 audio cases and eight content domains, the study highlights the importance of selecting the right STT provider for voice agents by focusing on transcription accuracy and its impact on downstream responses. The evaluation used a comprehensive methodology that included assessing transcription similarity, critical entity recall, and answer equivalence, alongside real-time latency measurements. The study found that while all models were closely matched in accuracy, OpenAI's gpt-4o-transcribe emerged as the top choice, offering the best balance of answer quality and low latency. It emphasized the value of using domain-specific vocabularies and post-transcription corrections to enhance the performance of STT systems, particularly for structured tokens like IDs and callsigns, which prove challenging for many models. Additionally, the research underscored the need to incorporate audio review in the evaluation process to differentiate between genuine errors and reference issues, suggesting that the choice of model should align with project-specific priorities, whether it be accuracy, speed, or the preservation of critical information.

Brainstore, a database tailored for handling agent traces, faced challenges with traditional phrase search in large datasets, which often resulted in slow queries due to common terms with rare intersections. The solution involved implementing shingled bloom filters using trigrams instead of unigrams, improving segment elimination by focusing on rare three-word combinations rather than individual common words. This approach significantly enhanced search efficiency, allowing for faster phrase search by pruning irrelevant data more effectively. The improved method was tested on real customer data, reducing the scanned data size from over 100 GB to less than 4 GB, resulting in a 25x increase in efficiency. As Brainstore continues to develop, it aims to further optimize its search capabilities and handle larger datasets, ensuring that agent debugging remains fast and efficient even as data volumes grow.

In preparation for the USA vs Belgium Round of 16 match in the 2026 FIFA World Cup, a detailed evaluation was conducted using the monolithic-pro configuration of a research tool, which was previously identified as cost-effective and efficient for football match analysis. This setup, which provides comprehensive mapping at a lower cost compared to other configurations, focuses on key elements like squad structure, player availability, recent form, and historical team relationships. Notably, the research highlights Belgium as the favorite with a 68% predicted chance of victory, emphasizing the tactical roles of key players such as De Bruyne and Lukaku, despite some injury concerns. The tool's analysis shows that the U.S. team faces challenges due to potential player availability issues and historical disadvantages. The research underscores the importance of inspecting and updating underlying evidence to ensure accuracy, as demonstrated by the update on Folarin Balogun's red-card status, which was lifted prior to the match. The study ultimately offers a clearer pre-match picture, leveraging insights from previous evaluations to enhance understanding and prediction accuracy.

During the 2026 World Cup, the Braintrust team explored the effectiveness of Parallel's web research agents in automating the data collection and analysis typically done by analysts and fans when examining football matchups. They utilized Parallel Web Systems’ Task API to create structured, source-backed maps of squad dynamics, including player availability, head-to-head records, and recent performance, which were then organized into knowledge graphs for easy inspection. The research experiments involved running 48 matchups through six different configurations, using two architectures—monolithic and fan-out—across three processing tiers to assess how research depth and task design interact. The findings suggested that while the monolithic-pro configuration offered a cost-effective solution with substantial coverage, the fan-out approach excelled in domain-specific tasks like injury analysis. Despite improvements in research depth, prediction calibration remained consistent across configurations, indicating that while the process provided a detailed and inspectable overview, it did not necessarily enhance forecast accuracy. The study demonstrated the utility of graph-based structures in making football intelligence evaluable, allowing for ongoing application throughout the tournament as team compositions and conditions evolved.

LLM observability in TypeScript requires a comprehensive tracing setup that provides a TypeScript SDK, supports various runtimes, and offers detailed insights into request paths and failures. Effective tracing should capture inputs, outputs, latency, and errors at each step, allowing teams to isolate issues in model calls, tool invocations, and runtime operations. Auto-instrumentation can simplify the integration of tracing into TypeScript applications, while manual instrumentation offers more control. The Vercel AI SDK, enhanced by Braintrust, enables detailed tracing and telemetry, facilitating debugging and evaluation of production traces. This setup supports not only model and tool call tracing but also the transformation of production traces into reusable evaluation datasets. By capturing comprehensive request data, teams can utilize these traces for testing and improving future releases, ensuring that any production failures identified are addressed before they affect users again. For optimal results, teams should select tracing tools that align with their app's runtime, framework, and quality assurance processes.

Traditional Application Performance Monitoring (APM) tools like Datadog, Grafana, and Honeycomb often fall short in effectively monitoring Large Language Model (LLM) applications, as they focus primarily on metrics such as latency, error rates, and system health, which may not reflect the quality of the model's output. OpenTelemetry offers a solution by integrating structured telemetry at the LLM layer, capturing detailed data about prompts, retrievals, tool calls, and model responses, which standard APM tools typically overlook. By implementing OpenTelemetry's GenAI semantic conventions, organizations can trace and evaluate LLM applications more effectively, ensuring output quality and system observability are interconnected. This approach allows teams to route spans to multiple backends, such as Braintrust for output scoring and traditional APM tools for operational monitoring, without needing to modify existing telemetry paths. Through distributed tracing and the use of both automatic and manual spans, OpenTelemetry provides a comprehensive view of LLM workflows, enabling teams to debug, assess quality, and turn production failures into test cases, thereby maintaining robust and reliable LLM application pipelines.

The text discusses the importance and methodology of tracing in Python applications, particularly those involving large language models (LLMs), to ensure smooth and efficient operation. It emphasizes the need for a comprehensive trace that captures various steps such as retrieval, preprocessing, model calls, and more, enabling teams to identify and resolve issues efficiently. The use of OpenTelemetry is highlighted as a foundational tool that provides a standardized approach for creating and transporting spans, but with additional LLM-specific interpretation required for effective debugging and evaluation. Braintrust is presented as a robust solution for Python-native instrumentation, allowing teams to trace LLM applications by automatically capturing inputs, outputs, latency, and costs, while also integrating with existing OpenTelemetry setups. It supports both auto and manual instrumentation, offering flexibility in tracing provider and framework calls as well as application-specific logic. Additionally, the text explains how traces can be transformed into evaluation datasets to enhance future release checks, using real production cases to ensure continued application reliability.