September 2026 Summaries

TypeSafe AI’s Jev is a specialized classifier for yes-or-no evaluation tasks that returns probabilities rather than generated explanations, positioning it as a lower-cost, faster alternative to generative LLM-as-a-Judge systems for tasks such as determining whether an assistant response addresses a user’s question. In a test on 1,000 production turns from Opik’s Ollie assistant, Jev and GPT-4o-mini agreed on 89.7% of verdicts, while Jev cost $0.0285 versus $0.0993 per 1,000 traces and had substantially lower median latency, although the comparison did not establish which judge was more accurate because no human-labelled ground truth was used. Disagreements clustered around Jev’s uncertain probability range, suggesting that its confidence scores can identify cases for human review rather than relying on random sampling. Opik supports tracing Jev calls, using Jev as an online evaluation rule, and optimizing the wording of its Boolean evaluation question against labelled datasets, while emphasizing that teams should compare models on identical real traffic, collect labels, define the policy being judged, and retain generative judges when downstream users need written reasoning rather than a numeric decision.

Comet’s Opik platform is now available through Nebius AI Cloud Applications, marking the first stage of a broader collaboration intended to bring AI agent observability and evaluation closer to the infrastructure used to train and run AI systems. Nebius provides an AI-native cloud platform spanning data centers, hardware, orchestration, managed inference, and agent infrastructure, while Comet adds experiment management, monitoring, and evaluation capabilities. Opik is an open-source platform that records LLM calls, agent traces, inputs, outputs, and intermediate steps, enabling teams to investigate agent behavior, compare model versions, and identify regressions across single- or multi-agent systems. Its evaluation framework supports custom metrics, automated scoring, and benchmarking alongside training and inference workloads. Nebius customers can also access Comet’s wider MLOps offerings, including experiment tracking, model versioning and registry functions, and production monitoring, supporting the AI lifecycle from fine-tuning through deployment.

Comet Cost Intelligence explored whether coding-agent sessions could be represented as recognizable phases and tasks to help organizations understand token spending driven by developer behavior, rather than only model settings or context bloat. After analyzing thousands of Claude Code sessions with a locally run Qwen3-4B model, the team found that sessions could be segmented into recurring activities such as exploration, implementation, validation, and discussion, while also revealing that multiple unrelated tasks within one session can create unnecessary context costs. Small models could make useful semantic judgments but struggled with long-range bookkeeping, so the system processes compressed transcript windows, carries short contextual notes between them, and uses deterministic rules to stitch results together, with validation and versioning intended to ensure stable, reproducible charts. The approach runs continuously on a single GPU at a projected capacity of roughly 1,700 to 2,000 active developers, and its development prioritized proving the underlying concept on real data before finalizing interface design. The authors acknowledge important limitations, including the absence of ground truth for phase labels, difficulty distinguishing developer prompts from agent self-prompting, and uncertainty about whether users unfamiliar with the system will interpret session shapes correctly.

Prompt management treats the instructions and configurations behind LLM applications as versioned production assets, combining messages, model settings, tools, parameters, and response schemas so teams can track changes, test candidates, deploy updates independently of application releases, and quickly roll back problems. It is especially important for agents because small wording changes can alter tool selection, retry behavior, workflow execution, costs, and structured outputs, although versioning alone cannot guarantee reproducible results when underlying models, retrieval indexes, or external tools change. Effective systems centralize prompt registries, preserve immutable diffs and ownership metadata, support environment promotion and playground testing, and, most importantly, connect versions to evaluations and production traces so regressions can be linked to specific changes. The comparison highlights platforms with different strengths, including open-source options such as Opik, Phoenix, Langfuse, and MLflow; managed evaluation-oriented systems such as Arize AX, Braintrust, and Galileo; and ecosystem-focused tools such as LangSmith and W&B Weave. The central recommendation is to choose a platform based not merely on its ability to store prompt versions, but on how well it integrates evaluation, observability, deployment, and the organization’s existing development workflow.

TrueFoundry AI Gateway can mirror its OpenTelemetry traces to Comet’s Opik platform, allowing teams to inspect and evaluate LLM requests—including prompts, outputs, token usage, latency, costs, model calls, and tool calls—without modifying application code. Since the gateway sits in front of model traffic and supports more than 1,000 LLMs through an OpenAI-compatible API, gateway-level tracing provides centralized visibility across services while retaining default trace storage in TrueFoundry. Setup requires an Opik API key, workspace, and project, followed by enabling the HTTP OTLP traces exporter in the TrueFoundry dashboard with Opik’s endpoint and required authorization, workspace, and project headers; metrics export should remain disabled because Opik accepts traces rather than OTLP metrics. Once configured, requests appear in Opik as chat-completion traces or nested agent traces, enabling end-to-end debugging, production-based evaluations, and analysis of slow or costly calls.