When Your System Is an Agent, You Need a Different Benchmark

Qodo's code review system evolved from a simple, single-command prompt to a sophisticated multi-agent architecture, presenting a challenge in maintaining accurate benchmarks. Initially, the system utilized a single LLM call to return code suggestions in a YAML format, which was straightforward to measure. However, as the system expanded into a multi-agent pipeline incorporating specialized agents for context collection, issue finding, and compliance enforcement, the original benchmarking method became inadequate. The new architecture required a shift in evaluation strategies to account for the complexity and non-determinism of the multi-agent system. This led to the development of a new benchmarking infrastructure using synthetic pull requests and LLM-as-Judge with ensemble voting to ensure precise evaluation of agent performance. By focusing on precision and recall across agents and utilizing ensemble judges, Qodo improved its ability to diagnose and address system failures, transforming the evaluation process from a static leaderboard metric to a dynamic, interpretable feedback loop. This methodological shift not only enhances the system's reliability but also provides a framework for other teams to evaluate multi-agent systems effectively.