May 2026 Summaries

The process of improving AI agents involves enhancing not just the prompts but the entire harness surrounding the model, which includes the tools it can call, the context it receives, the traces it emits, and the evaluations (evals) it runs. A continuous improvement loop allows these elements to be refined over time by tracing each run, evaluating specific spans, inspecting failures, and determining whether the agent or evaluator is incorrect. Through a live demonstration with Aakash Gupta, Arize AI cofounder Aparna Dhinakaran illustrated this workflow using a product management (PM) agent for Arize Phoenix, which processed GitHub data to generate a report, though initial success was limited by a lack of detailed understanding of decision-making processes. The key to agent improvement lies in a systematic approach of tracing, evaluating, debugging, refining, and repeating the loop, ensuring that failures are analyzed to identify specific areas for enhancement. Observability and traceability are emphasized as critical components, serving as inputs to the improvement loop, allowing teams to understand the trajectory of agent behavior and make informed adjustments. Ultimately, the harness—comprising context, tools, state, retries, routing, memory, evals, and review gates—determines the agent's effectiveness and capacity for self-improvement, highlighting the importance of a structured engineering methodology in developing reliable and adaptable AI systems.

The text discusses the challenges and solutions of automating the feedback loop in LLMOps (Large Language Model Operations) for AI agents, emphasizing the need for continuous evaluation and improvement after deploying LLM-powered applications. It introduces the Arize AX Airflow Provider, an open-source tool that integrates with Apache Airflow to streamline the orchestration of AI evaluation workflows, offering operators and sensors for tasks like dataset refresh, drift detection, and CI/CD gates. This integration allows AI systems to learn from production experiences by automating the process of capturing, evaluating, and improving system behavior based on production traces. The provider enables efficient management of the evaluation lifecycle, transforming observability data into actionable insights, and ensuring that system improvements are systematically integrated into production, thereby reducing manual intervention and enhancing reliability. The text also highlights the importance of open-source solutions in creating scalable and auditable AI operations and provides practical examples of how the Airflow provider can be used to automate and optimize LLMOps processes.

In an effort to reduce costs and improve efficiency, the author explores the use of smaller, local language models (SLMs) as alternatives to frontier models for specific AI tasks within a social and news app named Mima. By employing capability evaluations (evals) and prompt engineering, they successfully deploy a local 3B model that matches the performance of the Claude Sonnet model for summarization tasks while running faster and incurring no additional call costs. The text details the process of selecting a Small And Good Enough (SAGE) model through a methodical four-step framework that includes prototyping with a state-of-the-art model, setting success criteria, testing a range of models, and choosing the most suitable one based on a balance of accuracy and latency. The author highlights the importance of evals in assessing model capabilities and emphasizes the role of prompt engineering in mitigating issues such as hallucination rates. The use of deterministic solutions and engineering techniques to enhance model performance without additional inference costs is also discussed, alongside the implementation of regression evals to maintain model output quality over time.

LLM-as-a-Judge is an evaluation framework where language models assess outputs from other models based on predefined criteria, offering benefits for scaling evaluations beyond manual review. It is critical that the evaluation criteria are clearly defined, including the target quality, inputs, allowed outputs, decision rules, and practical examples, to avoid failures common in ambiguous criteria. The framework distinguishes between code evaluators and LLM judges, where code is used for deterministic checks and LLMs for semantic evaluations. The process involves using Boolean, categorical, or ordinal labels for clarity, avoiding open numeric scores unless justified. Calibration against human labels is essential to ensure agreement with human judgment, and explanations from judges should be treated as debugging aids rather than ground truth. Evaluators should be integrated into the engineering loop, with results stored near execution records to facilitate inspection and improvement. Continuous monitoring is necessary to manage biases and drifts in the judge's performance over time, ensuring that LLM-as-a-Judge supports better engineering decisions rather than acting as a standalone metric.

Testing seven models within a consistent agent harness revealed that while model swaps might appear as simple configuration changes, they more closely resemble product migrations due to the impact on operational behavior. The study involved models such as Sonnet, GPT, and Gemini, tested on GitHub agent tasks using a fixed setup to ensure consistency. Although correctness across models remained relatively stable, ranging between 79.6% and 85.1%, significant differences were observed in operational metrics like latency, tool-call counts, and retry behavior. The findings emphasize that while final-answer quality might remain constant, the path to achieving that answer can differ significantly in terms of cost, efficiency, and reliability, underscoring the importance of evaluating both correctness and operational behavior before implementing model changes in production.

A self-improving AI agent can be developed by leveraging existing data from human corrections, captured as a context graph, without the need for retraining. This approach addresses the discrepancy often found between AI recommendations and human decisions based on institutional knowledge not documented in formal policies. By capturing and analyzing human overrides in a structured context graph, patterns emerge that can be fed back into the AI system to improve its decision-making process. A procurement agent demo illustrates this, where human reviewer Vera Fye's overrides were used to enhance the agent's accuracy from a 53.8% to an 83.1% match rate over four cycles, solely by integrating structured human feedback. This method shows promise in various applications where human judgment complements AI, enabling continuous improvement without altering the agent's source code. The broader implication is the potential to harness human-AI disagreement as a valuable signal rather than discarding it as noise, thus enhancing agent performance in alignment with real-world complexities and institutional knowledge.

Coding agent tracing and evaluation is an open-source tool designed to enhance AI coding workflows by providing detailed insights into the operations of coding agents such as Claude Code, Cursor, Codex, GitHub Copilot, and Gemini CLI. This tool allows developers to meticulously inspect each step of a coding agent's process, including file reads, tool calls, command executions, retries, and token usage, thereby facilitating a deeper understanding of agent behavior and workflow efficiency. By collecting and analyzing trace data, developers can identify ineffective workflows, build reusable skills, and determine which coding models and prompts yield the best results, ultimately leading to systematic improvements in AI coding practices. Traces can be sent to platforms like Arize AX or Phoenix for further inspection and evaluation, enabling developers to experiment with different configurations and track their impact over time. This approach encourages the development of shared practices and reusable skills across teams, integrating coding agents into the broader software development stack with a focus on observability, evaluation, and iterative improvement.

Alyx is an AI agent designed to enhance the efficiency of developing other AI agents by streamlining the debugging and evaluation processes through advanced trace analysis tools. The developers of Alyx use the agent itself to build and improve Alyx, employing its capabilities to handle the dense and complex data within traces that are otherwise impractical to analyze manually. Key functionalities include regex-based search tools, structured JSON queries, and aggregation methods to identify patterns and categorize errors across multiple spans. This system allows for efficient debugging by separating the reporting of failures from the detailed analysis, enabling a small team to handle issues that would otherwise require significant time from many engineers. The approach of using Alyx to analyze its own performance helps identify gaps in the tool's capabilities, ensuring it is robust enough for external use. The development process emphasizes the importance of integrating efficient data analysis and categorization to optimize agent development workflows.

Over the past year, AI models have significantly improved in their ability to follow complex instructions, as demonstrated by the updated IFScale benchmark. This benchmark, originally detailed by Jaroslawicz et al. (2025), measures how well models can adhere to numerous constraints, such as including specific keywords in a business report. While older models struggled to maintain accuracy beyond 200-300 simultaneous instructions, current frontier models, like GPT 5.5 and Gemini 3.1 Pro, can now handle up to 5,000 instructions with high accuracy. This advancement has implications for AI engineering, allowing for more detailed prompts and reducing the need for compressed skill files, although it introduces new considerations regarding cost and processing time. Different models exhibit unique failure modes; for example, some models politely refuse complex tasks, while others overthink or misinterpret constraints. Despite these challenges, the ability to manage extensive instructions opens new possibilities for developing sophisticated AI applications.

Arize Phoenix is evolving from an observability tool to a context platform designed to facilitate collaboration between humans and AI agents in software development. As AI agents increasingly write and modify code, they require more than traditional human-centered observability tools; they need access to traces, evaluations, feedback, and other contextual data that can be queried and acted upon. This shift acknowledges that while agents generate changes, they must also verify whether these changes improve system behavior, using programmatic interfaces like APIs and CLIs. Phoenix aims to provide this context, enabling agents to self-improve by tracing, evaluating, diagnosing, fixing, and rerunning their tasks. The platform supports both active and passive interaction modes, where humans either direct agents or let them autonomously propose changes, thus balancing oversight and autonomy based on trust. This transformation underscores a future where agents can debug other agents, while humans make critical decisions about delegation. Phoenix also introduces Phoenix Intelligence, which includes an assistant for human collaboration and expert agents for continuous evaluation, reinforcing the platform's role in enhancing AI-native software development.

In this exploration of agent harness architecture, the author discusses the evolution and challenges of building harnesses capable of adapting to multiple model generations, particularly focusing on the implicit finish assumption within Claude Code's harness and its implications when applied to different models like OpenAI's GPT series. The piece outlines how the Claude Code harness, which assumes task completion when no tool calls are made, led to issues with models like GPT-4o that separate narration and action, resulting in premature loop exits. To address this, the author at Arize developed an Explicit Finish and an Adaptive Finish harness, with the latter proving to be more reliable in catching false finishes while maintaining efficiency. Through comprehensive testing across different models and tasks, the author highlights the importance of adaptable design and thorough evaluation to ensure harness reliability, emphasizing the need for eval suites that can adapt to model updates and task variations. The article concludes with a reflection on the necessity of understanding and configuring hidden tunings within harnesses to prevent false assumptions and improve performance across various model behaviors.

Testing and debugging AI agents, such as Alyx, require distinct approaches compared to traditional software due to the non-deterministic nature of AI outputs. When developing Alyx, the team discovered that small changes in prompts could lead to unpredictable failures, necessitating a robust testing framework. They moved from inefficient manual testing to using real production traces as test cases, capturing actual user interactions to ensure comprehensive evaluation of agent behavior. By employing LLM-based evaluators, they could assess whether outputs met expectations without relying on exact matches that are prone to brittleness. This method allows for more flexible and accurate testing, as it focuses on understanding the intent behind outputs. Additionally, integrating these tests into CI/CD pipelines ensures continuous monitoring and quality control, preventing regressions. Experimentation over time, especially during model upgrades, helps track performance trends and catch anomalies early. Ultimately, the testing framework fosters better communication within teams by providing a shared language for evaluating AI behavior, thus enhancing collaboration and reducing ambiguity.

Swarm management, a concept integral to advanced AI systems, is emerging as a critical challenge in managing fleets of long-running agents, beyond merely spawning subagents. The article highlights the need for a robust infrastructure that governs the lifecycle, identity, and completion of these agents, exemplified by the OpenClaw system. OpenClaw's architecture utilizes session keys, run IDs, and a push-based completion model to manage agent swarms, ensuring that each agent's state and outputs are effectively tracked and managed across various operational scenarios. This system highlights the necessity for concurrency, queue policies, and recovery processes to maintain control over complex agent networks. Unlike simple delegation systems, which often treat completion as a return value, swarm management requires a sophisticated control plane that ensures agents are properly directed, interrupted, and terminated when necessary, emphasizing roles and runtime safety mechanisms to prevent unregulated growth and ensure scalability. Ultimately, swarm management extends beyond traditional agent frameworks, evolving into a comprehensive runtime infrastructure capable of coordinating a diverse array of tasks across multiple agents.

An evaluation harness is a standardized infrastructure designed to improve the evaluation process of AI systems by transforming it from isolated, manual assessments into a scalable and repeatable system. It operates as a three-stage pipeline that defines what is evaluated, how it is scored, and what actions are taken based on the results, making it crucial for the production and continuous improvement of AI applications. Unlike traditional benchmark runners that focus solely on model performance against static datasets, an evaluation harness evaluates live execution data across multiple dimensions, such as spans, traces, trajectories, and sessions, using diverse scoring methods and triggering subsequent actions like alerts, CI/CD gates, and annotation queues. This comprehensive approach is essential for modern AI systems, such as agents and RAG pipelines, which require ongoing evaluation to maintain quality and reliability in production environments. Platforms like Arize provide tools to implement evaluation harness workflows, enabling teams to integrate evaluation into their development and operational processes effectively.

In the evolving landscape of enterprise AI agents, the need for standardized agent telemetry has become crucial as these agents transition from demos to production workflows, demanding transparency in their operations. The collaboration between Google Cloud and Arize AI aims to address this challenge by promoting a unified telemetry model through OpenTelemetry and OpenInference, ensuring that agent behavior can be consistently captured, analyzed, and improved across different platforms without the constraints of vendor-specific systems. This approach promises to enhance observability by turning traces into structured, queryable timelines that illuminate the decision-making paths of agents, thus facilitating debugging, compliance, and iterative improvements. As a result, teams can experiment more rapidly, migrate seamlessly between tools, and maintain a robust audit trail, ultimately leading to a more reliable and flexible production environment for AI agents.

An evaluation comparing MCP (Model Context Protocol) and CLI (Command Line Interface) skills for agents found that while MCP can be more expensive and slower on complex tasks, it offers benefits in enterprise environments due to its use of OAuth for authentication and its potential for agent integration. The study used Claude Opus 4.6 to perform GitHub tasks across different difficulty levels, testing both MCP and CLI approaches. MCP struggled with tasks that required compositional logic, resulting in higher costs and latency, while CLI skills demonstrated efficient command execution due to their ability to compose commands. However, the evaluation also revealed that bare CLI, without any specific skill instructions, performed robustly, leveraging extensive pre-existing training data. The findings suggest that the debate over MCP versus CLI is misguided, advocating instead for a hybrid approach where each is used according to the specific requirements of the task at hand, such as local automation or remote enterprise deployments. The evaluation underscored the flexibility and adaptability required in choosing between MCP and CLI, highlighting that the choice should depend on the context of use rather than a direct comparison of capabilities.