Same Model, Different Harness

A 8090 Labs study evaluated seven coding models across five agent harnesses on 113 real-world software tasks, finding that the harness—the system controlling prompts, tools, file edits, and recovery from failures—can substantially affect performance, reliability, and cost. With model settings held constant, GLM 5.3 Flash varied from a 24% pass rate in Codex to 59% in DeepSeek’s harness, while GPT-5.6 Sol changed by only 6 points, showing that harness sensitivity differs by model. No model performed best in the harness made by its own developer, and Claude Fable 5.1 notably improved from 41% in Claude Code to 61% in Codex, though it used more tool calls there. Repeated runs also showed that average success rates can conceal inconsistent task completion, while cost per successful result sometimes favored a more expensive individual run with a higher pass rate. Results could further vary by programming language, although language-level samples were smaller. The researchers conclude that teams should assess models and harnesses as combined systems, evaluate repeatability alongside average performance, and compare cost per success rather than token or run prices alone.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.