RL training for math reasoning

The report describes efforts to develop reinforcement-learning infrastructure and train mathematical reasoning models using Group Relative Policy Optimization (GRPO), finding that RL can substantially improve performance beyond supervised fine-tuning when models begin with strong long chain-of-thought capabilities. The team selected NVIDIA’s NeMo Aligner as a short-term platform because of its feature completeness and support, while planning a longer-term move to the simpler, internally maintainable torchtune framework; it also improved rollout throughput with vLLM integration and resolved major log-probability alignment, memory, and implementation issues. Experiments on Llama 3.1 8B models and math datasets including GSM8K, MATH, ORZ, NuminaMath, and MATH-500 showed that an ORZ-supervised warm-up followed by GRPO achieved roughly 0.70 pass@1 on MATH-500, compared with about 0.38 for the original instruct model and 0.60 for the SFT model. Results indicated that varied training difficulty, appropriate learning rates, moderate sampling temperature, and prior exposure to long reasoning traces strongly affect training efficiency and stability. The work also identified self-repetition and maximum-length collapse as important failure modes, especially when format rewards or overly aggressive learning rates were used, while noting that GRPO’s group-based advantage normalization can create zero-gradient cases and favor easier questions. Future work will examine curriculum-learning strategies, revised reward and advantage calculations, and continued infrastructure and algorithm improvements.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.