Introducing FrontierHarness Eval

FrontierHarness Eval compares how nine production coding harnesses, across 12 configurations, influence agentic coding-task performance, cost, speed, and caching when using the same Kimi K3 model, Runta runtime, and 30 verifier-scored Terminal-Bench and DeepSWE tasks. Across 360 fresh, checkpoint-restored trials, Codex achieved the highest pass rate at 66.7%, Pi offered strong value at $2.43 per pass, Exo Harness had the lowest cost at $1.05 per pass despite a 53.3% pass rate, and DSH Minimal was fastest with a 5-minute-41-second median runtime; DSH Creator ranked competitively across all three measures. The evaluation found that pass rates varied less than costs, failures could be expensive, cache-hit rates did not necessarily indicate low spending, and model-vendor lineage did not confer a clear advantage, as Kimi Code placed near the middle. Claude Code matched DSH Creator’s 63.3% pass rate but showed substantially higher costs, potentially due to poor interaction among its caching approach, Kimi K3’s implicit cache behavior, and the gateway’s Messages API path rather than the harness alone. The benchmark acknowledges limits including its software-engineering focus, one-attempt design, harness-specific turn limits, and inability to isolate harness effects from model and gateway behavior, with future plans to test more tasks, harnesses, and complete harness-model compatibility matrices.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.