Agent-as-a-Judge: Evaluate Agents with Agents

The "Agent-as-a-Judge" framework presents an innovative approach to evaluating AI systems, addressing limitations of traditional methods that focus solely on final outcomes or require extensive manual work. This new paradigm uses agent systems to evaluate other agents, offering intermediate feedback throughout the task-solving process and enabling scalable self-improvement. The authors found that Agent-as-a-Judge outperforms LLM-as-a-Judge and is as reliable as their human evaluation baseline.