How to build a better agent harness with traces and evals

The process of improving AI agents involves enhancing not just the prompts but the entire harness surrounding the model, which includes the tools it can call, the context it receives, the traces it emits, and the evaluations (evals) it runs. A continuous improvement loop allows these elements to be refined over time by tracing each run, evaluating specific spans, inspecting failures, and determining whether the agent or evaluator is incorrect. Through a live demonstration with Aakash Gupta, Arize AI cofounder Aparna Dhinakaran illustrated this workflow using a product management (PM) agent for Arize Phoenix, which processed GitHub data to generate a report, though initial success was limited by a lack of detailed understanding of decision-making processes. The key to agent improvement lies in a systematic approach of tracing, evaluating, debugging, refining, and repeating the loop, ensuring that failures are analyzed to identify specific areas for enhancement. Observability and traceability are emphasized as critical components, serving as inputs to the improvement loop, allowing teams to understand the trajectory of agent behavior and make informed adjustments. Ultimately, the harness—comprising context, tools, state, retries, routing, memory, evals, and review gates—determines the agent's effectiveness and capacity for self-improvement, highlighting the importance of a structured engineering methodology in developing reliable and adaptable AI systems.