From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding - The JetBrains Blog

JetBrains researchers propose an evaluation framework for agentic coding systems that supplements resolve rate, or the percentage of benchmark tasks solved, with measures of execution efficiency, patch quality, and process quality drawn from agent trajectories. Using Junie to compare models across coding benchmarks, they found that similar or identical task outcomes can mask substantial differences in cost, number of steps, repository exploration, validation behavior, patch scope, redundancy, and hallucinated dependencies or APIs. Claude Opus 4.7 showed stronger diagnosis and more contained patches but sometimes failed to complete or validate fixes, while Gemini 3.5 Flash more frequently ran executable checks but often took longer paths, repeated work, and produced broader or less grounded changes. A wider comparison found GPT-5.5 had the highest resolve rate and consistently validated changes, Opus led patch-quality measures, and Qwen 3.6 27B FP8 offered much lower cost at lower task success. The authors emphasize that these profiles depend on the specific agent setup and that LLM-judge assessments are imperfect, arguing that detailed evidence about how agents work can improve model selection, prompting, and agent design beyond leaderboard rankings alone.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.