When does distillation help reinforcement learning?

Researchers evaluated whether supervised distillation on verified GPT-5.6-Sol reasoning traces improves later reinforcement learning with GRPO across Qwen3 models ranging from 0.6B to 8B parameters and sixteen verifiable reasoning tasks. Although fine-tuning on teacher solutions improved pre-RL accuracy at every size, its post-RL value declined with model scale: after 80 million RL response tokens, distilled models outperformed RL-only models at 0.6B and 1.7B, showed only a small advantage at 4B, and underperformed at 8B. Increasing the amount of teacher data from 0.5M to 4M tokens also did not reliably improve final RL performance, with the best dose varying by model size and larger models sometimes performing best without distillation. Distillation was more helpful on structured tasks, such as those involving conventions or spatial reasoning, while base models generally matched or surpassed distilled variants on calculation tasks. Analyses suggest that SFT can help smaller models by increasing the likelihood of generating correct responses and therefore supplying more useful RL training signals, but may reduce reward variation for larger models when response groups become too often entirely correct; it also consistently reduced next-token entropy, though entropy alone did not explain final accuracy. Overall, the findings indicate that distillation is not universally beneficial before RL and should be selected according to student size, task type, training budget, and the specific RL procedure.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.