What we learned testing 7 models under the same agent harness

Testing seven models within a consistent agent harness revealed that while model swaps might appear as simple configuration changes, they more closely resemble product migrations due to the impact on operational behavior. The study involved models such as Sonnet, GPT, and Gemini, tested on GitHub agent tasks using a fixed setup to ensure consistency. Although correctness across models remained relatively stable, ranging between 79.6% and 85.1%, significant differences were observed in operational metrics like latency, tool-call counts, and retry behavior. The findings emphasize that while final-answer quality might remain constant, the path to achieving that answer can differ significantly in terms of cost, efficiency, and reliability, underscoring the importance of evaluating both correctness and operational behavior before implementing model changes in production.