The hidden mismatch: Evaluating coding agents on the work developers actually do

Analysis of seven weeks of internal developer traffic found that public coding benchmarks such as SWE-bench, HumanEval, and Spider poorly represent real agent-assisted development, where only about 11% of turns request code generation and many involve clarifying questions, iterative debugging, pasted context, reviews, confirmations, and git-related requests within multi-turn sessions. Real developer messages are typically short, ambiguous, and embedded in ongoing work with prior conversation history and modified files, while benchmarks usually present isolated, well-specified tasks in clean environments and overemphasize difficult architectural problems. To address this gap, the authors created a 613-task benchmark using traffic-derived task quotas, public and internally solved repository tasks, developer-like prompt wording, automated tests, and replayed mid-session contexts. When evaluated through an intelligent routing gateway, the new benchmark closely matched live traffic’s premium-model routing rate, landing within 0.5 percentage points of a 25% target, whereas a public benchmark caused routing near 50%. The findings argue that organizations should evaluate coding agents using distributions, language, difficulty, session context, and internal codebases that reflect their actual workflows, and should assess routing and cost decisions at the session level rather than relying on isolated benchmark scores.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.