October 2026 Summaries

No summary generated yet.

No summary generated yet.

No summary generated yet.

No summary generated yet.

No summary generated yet.

Braintrust has introduced Coding Agent Insights in public preview to help teams trace, inspect, and reduce the costs of coding-agent sessions by identifying inefficient patterns such as incorrect test commands, repeated failed edits, or missing repository instructions. The dashboard tracks token use and costs across developers, models, tools, branches, and API keys, while allowing users to inspect individual sessions or ask the Loop assistant to investigate activity in natural language. Teams can schedule analyses that detect recurring issues, organize sessions with Topics, review findings in Patterns, and use supporting evidence to improve agent instructions, skills, or tools. The product emphasizes measuring results through comparable tasks, focusing on reduced failed attempts and model costs rather than aggregate weekly spending alone. It supports tracing integrations for Claude Code, Codex, OpenCode, Pi, Antigravity, and Grok, and is available to Braintrust SaaS and BYOC customers, with self-hosted availability beginning with dataplane 2.13.

Production AI agent failures may require investigation even when runs complete without errors, particularly when customers dispute claimed actions, releases alter behavior, or compliance rules may have been violated. Agent traces provide connected evidence of model calls, tool arguments and results, errors, timing, and metadata such as user, session, prompt, and model versions, enabling investigators to reconstruct what the application recorded and compare the agent’s final response with the outcome of its tools. However, traces cannot prove uninstrumented actions, reveal masked data, recover sampled or expired records, or confirm that external systems persisted a requested change, so consequential actions must be verified against systems of record. Braintrust supports searches by tool name, text, metadata, time, and span-level conditions to locate individual executions, inspect them through hierarchical, chronological, and timeline views, and identify retries, loops, latency, and recovered errors. After validating a failure pattern in one trace, teams can query production-wide records, distinguish calls from affected runs, group results by users or releases, account for incomplete data, and save findings for shared incident response.

Production LLM traces capture an application request as a tree of connected spans, preserving model calls, tool executions, prompts, outputs, identifiers, and metadata needed to reproduce and investigate failures. Because agent traces can contain large prompts, retrieved documents, and tool outputs, storage systems must balance completeness, cost, searchability, and retention; truncation and sampling can reduce volume but may remove critical evidence, while separating indexed metadata from accessible attachments can preserve large payloads. Effective investigations require full-text search, structured span filters, consistent application metadata, and the ability to distinguish conditions occurring within the same span from those occurring elsewhere in a trace. Organizations can store raw records in object storage, but doing so requires additional indexing, query, and trace-reconstruction tools, whereas dedicated trace backends provide integrated search and inspection, often alongside archival exports. Evaluations should test complete trace retrieval, search speed, large-payload handling, retention boundaries, exports, indexing freshness, and access controls using realistic production workloads. Braintrust presents Brainstore as a trace database built on object storage with indexing for semi-structured AI data, supporting trace search, attachments, configurable retention, exports, role-based access, and visual inspection of span trees, timelines, conversations, and raw records.

Reliable LLM document-extraction evaluation requires field-level scoring rather than a single aggregate accuracy measure, because errors in critical values such as totals, dates, identifiers, and tax details can make an otherwise high-scoring record unsafe for automated processing. Evaluation datasets should preserve the source document, expected labels, model output, and stable metadata, with either one row per field for simple field measurements or one row per document when acceptance depends on multiple fields passing together. Comparison methods must match downstream requirements: free text can use normalized similarity measures, dates should be canonically parsed, financial totals generally require exact decimal agreement, and identifiers must preserve meaningful characters such as case and leading zeros. Normalization and tolerances should only reflect equivalences already accepted by production systems, while omitted values, legitimate absences, invented values, and illegible sources should be recorded as distinct failure categories. Braintrust supports custom scorers, multiple named field scores, document-level acceptance checks, experiment comparisons, attachments for source review, and CI integrations, allowing teams to identify regressions in specific fields and enforce release policies that prevent changes from shipping when critical extraction requirements fail.

Raw AI conversation transcripts alone provide incomplete and biased product evidence because manual review covers only a small, selectively visible portion of traffic and often lacks metadata linking interactions to prompts, models, features, user segments, and eventual outcomes. Effective analysis begins with a specific product decision and requires structured tracing of sessions, configurations, feedback, behavioral signals, and downstream results, alongside privacy protections such as redaction, retention limits, and controlled access. Teams can classify conversations by intent and outcome using validated taxonomies, discover emerging issues through clustering, and compare trends across time, customer segments, product features, and model or prompt versions. Findings should be prioritized by frequency, severity, customer impact, and confidence, then investigated to distinguish engineering defects from prompt, retrieval, discoverability, or support issues. Validated patterns can inform roadmaps, support processes, prompt and model evaluations, regression datasets, and production monitoring, while Braintrust is presented as a platform for structured traces, automated topic analysis, human validation, and linking production failures to ongoing quality testing.

Lovable AI prompt or model updates should be evaluated for intended improvements and possible regressions before release, because changes to an existing Edge Function can affect live users even when frontend changes remain unpublished. The process uses Braintrust to log production requests, build datasets containing representative cases, known failures, and regression guards, then define separate scorers for quality goals such as summary faithfulness and preserved category accuracy. Builders can explore prompt or model variants in Braintrust Playground, but must validate promising changes in a separate candidate Edge Function through remote evaluations that test the full application path, including authentication, request handling, model calls, response parsing, latency, and errors. Comparing preserved Braintrust experiments for production and candidate endpoints against the same dataset enables review of individual regressions, intended fixes, reliability, and cost before promoting the candidate to production. The approach emphasizes secure non-blocking logging, correctly formatted test inputs, safeguards against side effects, matching experiment configurations, and ongoing expansion of evaluation datasets using newly observed production failures.

Useful AI agent traces enable engineers unfamiliar with an agent to reconstruct a run, identify why it produced an outcome, and isolate performance or reliability problems through clearly structured spans. Each meaningful model call, tool invocation, retrieval operation, retry, handoff, and state update should have stable descriptive names, appropriate types, consistent parent-child relationships, and preserved inputs, outputs, arguments, raw results, model context, configuration, timing, token usage, costs, and errors. The order-status example illustrates how complete tracing can distinguish a model’s incorrect selection of a US returns policy for an order shipping to Germany from a separate carrier-service timeout and retry. Searchable metadata such as user and session IDs, environment, request context, feature flags, and prompt, model, and agent versions helps teams find related production runs and compare behavior across releases. The guidance also emphasizes tracing custom and cross-service work, preserving context during parallel execution, recording failures even when retries or fallbacks succeed, masking sensitive data while retaining useful structure, and marking truncation or attachment limits. Braintrust’s trace viewer supports investigation through span hierarchies, timelines, conversational threads, raw records, filters, and debugging tools, while acknowledging that traces provide observable evidence around model decisions rather than complete access to internal reasoning.