How we benchmark AI agents and tools with Harbor and Arize Phoenix

Agent benchmarking is difficult because realistic predeployment evaluation requires reproducible tasks, data, tools, environments, and resettable state, while simplified mocks often omit critical frontend context, external services, and real database behavior. The Phoenix team encountered these limitations while testing its PXI coding agent and adopted Harbor, an open-source sandboxing framework that runs agents in fresh local or cloud environments and uses verifiers to evaluate outcomes. Harbor enabled testing PXI against a real Phoenix server and seeded data, while also supporting evaluations of MCP servers, skills, CLIs, and coding agents such as Claude Code and Codex using shared tasks and outcome-based measures of correctness and efficiency. A Phoenix plugin records Harbor tasks as versioned datasets, agent configurations as experiments, verifier results as annotations, and agent trajectories as traces, allowing teams to inspect tool use, model calls, failures, turns, and task-version changes. Although agents may achieve similarly correct answers, trace analysis can expose substantial differences in their workflows, helping developers refine prompts, skills, and tools to reduce unnecessary steps without reducing performance.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.