Content Deep Dive

How Much Does Reasoning Effort Affect Score and Cost?

Blog post from Vals

Post Details
Company
Date Published
Author
Derrick Liang
Word Count
1,612
Company Posts That Month
11
Language
English
Hacker News Points
2
Post removed?
No
Summary

An evaluation of seven frontier AI models across eight Vals Index benchmarks found that reducing reasoning effort often lowers costs far more than performance, though the trade-off varies by model and task. At their lowest settings, four models retained at least 80% of their leaderboard-level score for half the cost or less, with Claude Opus 5.5 and Grok 4.7 each using only 16% of the cost while retaining 80% and 86% of performance, respectively. Across 22 of 56 model-and-benchmark comparisons, a lower-effort setting matched the higher-cost leaderboard setting, while no lower setting significantly exceeded it. Retrospectively selecting the cheapest highest-scoring setting for each task improved scores by a median of 6 points while costing 64% of the leaderboard configuration, although this represents an upper bound because real systems must select effort before knowing outcomes. Additional reasoning was especially valuable for Legal Research, Code Migration, Terminal-Bench, tax, and coding tasks, but produced comparatively small gains on finance, HLAB, and EMB benchmarks. The analysis notes limitations including testing only about half of each benchmark’s tasks, limited repeated runs, possible provider changes over time, and statistical uncertainty in small performance differences.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Gemini 4 Argon 8 123 33 20 -64%
Opus 5.5 8 1,073 88 45 +72%
GPT-6 Astra 6 190 53 30 -42%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.