September 2026 Summaries

Confident AI has introduced Jev-as-a-Judge, a feature intended to make AI evaluation and classification inexpensive and fast enough for teams to score a much larger share of production traffic rather than relying on small samples. Powered by TypeSafe AI’s Jev System One decision model, the service handles bounded tasks such as determining whether a response is supported or assigning labels, while language models remain responsible for generative work. The company says Jev costs $0.042 per million input tokens, typically returns decisions in about 100 milliseconds, and produces consistent verdicts with probability scores, though it does not provide written reasoning. Greater evaluation coverage is positioned as improving regression detection, sentiment and category correlations, customer monitoring, and early detection of issue spikes. Jev-as-a-Judge is available through Confident AI project model settings and in the open-source DeepEval framework.

Confident AI has introduced Customer Monitoring, a product analytics feature for B2B AI agents that organizes trace, evaluation, and classifier data around customer accounts rather than only individual users. The feature provides account-level views of cost, activity, sentiment, recurring issues, use cases, and user-level details, while allowing teams to inspect the underlying traces and conversations behind each metric. By connecting product analytics directly to production evaluations, it aims to help product managers identify account health and usage patterns, customer success teams prepare for customer concerns, and engineers diagnose issues such as retrieval failures or broken tool calls. Customers can be attributed to traces by adding customer and user information to the existing tracing context, with account dashboards populating as traffic arrives.

Confident AI has launched Category Correlation, a feature designed to identify relationships among classified production signals in LLM applications without requiring teams to manually filter through every possible label combination. Using classifiers for dimensions such as agent issues, sentiment, use cases, and security or safety events, the tool analyzes live traces and conversation threads to show which category pairs occur more or less often than expected. It provides an overall Cramér’s V score for the relationship between two classifiers, a heatmap of label-level associations, and direct access to the underlying traces or threads for investigation. The feature is intended to help product, engineering, and evaluation teams prioritize issues linked to user outcomes, diagnose behavior, and create relevant test cases, while emphasizing that correlation indicates areas for investigation rather than proving causation.

Confident AI has launched its Drift page, a production observability feature designed to automatically identify meaningful behavioral changes in AI agents and connect them to the specific agent configurations and traffic segments involved. Using OpenTelemetry-native trace data, the platform fingerprints components such as models, prompts, providers, endpoints, and tool use to create automatic configuration versions, enabling consistent comparisons across quality, sentiment, error, latency, and cost metrics. Drift distinguishes sudden anomalies from sustained regressions through statistical methods, including median absolute deviation and significance testing, rather than flagging ordinary chart fluctuations. It also analyzes segments such as model, provider, integration, metadata, retrieval settings, and classifier labels to reveal issues that aggregate metrics may conceal, ranking findings by both impact and affected traffic volume. The feature supports more than 25 integrations, including LangChain, LangGraph, Google ADK, and Amazon Bedrock AgentCore, and is now available on Confident AI.

Confident AI has launched Voice AI Evals, a feature intended to consolidate voice-agent testing, general AI evaluation, and production observability within one platform. The offering evaluates complete conversational experiences rather than transcripts alone, using persona-based simulations and challenging conditions such as interruptions, noise, and changing turn patterns. It introduces seven deterministic audio-focused metrics covering naturalness, intelligibility, consistency, turn-taking, responsiveness, audio integrity, and reliability, which can be combined with existing multi-turn LLM evaluation metrics to assess both task success and usability. The platform supports SIP, phone calls, WebRTC, WebSockets, and LiveKit integrations for agents and speech systems built with tools such as Vapi, ElevenLabs, and OpenAI. Confident AI positions the feature as a shared workflow through which engineers, QA staff, domain experts, and product teams can review calls, investigate failures, annotate outcomes, and compare performance across scenarios.

Confident AI has introduced confident-trace, a lightweight Python and TypeScript SDK for production monitoring of AI agents that replaces DeepEval as the recommended tracing option while leaving DeepEval focused on evaluation workflows. Built on OpenTelemetry, the SDK captures agent operations, model calls, tool use, and custom application code as spans, exporting them through OTLP directly to Confident AI or via an OpenTelemetry Collector. It offers automatic and custom instrumentation across 25 integrations, including LangChain, LangGraph, Pydantic AI, Vercel AI SDK, and Mastra, with optional framework-specific dependencies. The company presents a Pydantic AI quickstart and describes how production traces can help teams diagnose agent failures, convert real interactions into evaluation test cases, and validate improvements.