The Unreasonable Efficiency of a Single Gradient Step
Blog post from Baseten
Experiments replicating recent claims of nearly linear reinforcement-learning trajectories for language models find that extrapolating early training updates can outperform or closely match much longer RL runs on mathematical benchmarks, including scaling Qwen2.5-Math-1.5B’s first gradient update by 1,000 times to exceed a 500-step GRPO model on MATH. However, analysis of corrected examples indicates that much of this gain results from eliminating repetitive “doom-loop” behavior and failures to terminate rather than producing broadly new reasoning capabilities, with 72% of extrapolation-fixed MATH cases involving heavy repetition in the base model. Results varied substantially by task and scale: early-update extrapolation offered limited or no benefit for function calling and Knights & Knaves, while fitted rank-1 RELEX trajectories could recover some performance across several smaller environments when rescaled. At the larger Qwen3-8B scale, linear extrapolation failed to match full RL training, particularly on Knights & Knaves and IFEval, although moderate rescaling retained partial improvements. The findings suggest that low-dimensional structure in RL optimization may help cheaply address easier model failure modes, but does not consistently replace extended RL training, underscoring the importance of inspecting model outputs alongside aggregate metrics.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| LLM | 3 | 6,578 | 1,243 | 236 | +9% |
| Reinforcement learning | 2 | 162 | 88 | 37 | -7% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.