November 2026 Summaries

An experiment using Opik evaluates and optimizes an AI remediation agent for a Docker-based payments platform with realistic metrics, logs, alerts, and ten incident types, including bad deployments, fraud declines, stale FX rates, and false alarms. The agent, powered by GPT-5-mini, investigates incidents through monitoring and operational tools, but is assessed not only on root-cause identification and remediation actions, but also on efficiency and strict safety constraints such as avoiding unauthorized fraud-control changes or payment disruptions. Initial testing showed that the agent often found the correct solution but wasted tool calls and failed to submit a diagnosis, achieving an overall validation score of 0.34. Prompt optimization with Opik’s HRPO method improved diagnosis and efficiency, but initially introduced unsafe behavior after feedback inadvertently revealed expected solutions and encouraged overly decisive actions. After redesigning feedback, weighting safety-critical scenarios more heavily, and adding explicit prohibitions and pre-action safety checks, the final version achieved an overall score of 0.57 and perfect safety across thirty validation runs, though its prompt grew substantially and increased token and latency costs. The results are limited by synthetic incidents, a small validation set, and partial reliance on an LLM judge, but illustrate how tracing, repeatable evaluations, and safety-focused optimization can help develop more reliable infrastructure agents.

A practical example builds a two-agent customer-support system with Google’s Agent Development Kit and instruments it using Opik to capture agent handoffs, Gemini calls, tool calls, inputs, outputs, token usage, costs, and conversation threads. The setup uses a recursive Opik tracer for all agents and tools, while a wrapper around each customer turn keeps multi-agent handoffs within a single trace and associates turns with an ADK session thread. The project evaluates 23 challenging support scenarios, including prohibited refunds, prompt-injection-style requests, damaged products, vague requests, and multi-part questions, scoring required tool use, handoffs, refund-policy compliance, grounded order IDs, and answer relevance. It emphasizes that safety checks should inspect tool-call arguments in code rather than rely solely on an LLM judge, which incorrectly downgraded one valid refusal. In a one-trial comparison, Gemini 3.5 Flash-Lite matched Gemini 3.8 Flash on refund safety and handoffs while costing roughly five times less and responding about six times faster, though each model made different errors. The guidance extends to production use through session-based tracing, environment and model metadata, retries, version compatibility, and converting problematic production traces into future evaluation cases.