FP8 Reinforcement Learning in SkyRL: Preserving Policy Consistency Across Training and Rollout

SkyRL has added FP8 acceleration for reinforcement-learning training and rollout, including FP8 linear-layer computation, model weights, KV caches, and trainer-to-vLLM weight transfers, while retaining BF16 or FP32 for precision-sensitive operations and optimizer state. The work addresses a key RL reliability issue in which independently quantizing weights in the trainer and rollout engines creates numerically different policies, leading to unstable training behavior; SkyRL’s on-policy weight synchronization instead transfers the trainer’s FP8 weight payloads and scale metadata directly so rollout weights are bitwise identical after synchronization. In 400-step DAPO experiments with Qwen3.5-9B on eight H100 GPUs and Qwen3.5-35B-A3B on eight B200 GPUs, FP8 with this synchronization method closely matched BF16 reward, pass@8, and response-length trends while reducing maximum end-to-end step time by about 19% and 23%, respectively, largely through faster memory-bound generation. FP8 parameter storage also reduced per-GPU trainer weight memory by 39–42% on Hopper systems, although training-phase gains were limited by scale-management and host-dispatch overhead, and support varies by hardware and quantization recipe.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.