Training-Inference Parity in MoE Models: Where Numerics Drift

The blog post discusses the numerical drift issues encountered in Mixture-of-Experts (MoE) models like Kimi K2.5 and Qwen3.5-MoE, where kernel fusions that are mathematically equivalent can result in different numerical outputs due to the non-associative nature of floating-point addition. This drift poses challenges in maintaining training-inference parity, which is crucial for reproducibility, customer trust, and the integrity of RLHF/GRPO reward systems. The text highlights how the divergence arises from differences in FP accumulation orders across multiple layers and operations, such as all-reduce topologies, fused communication with computation, and multi-operation fusions in MoE models. These issues are exemplified through case studies showing how slight numerical differences can cascade through the network, particularly in MoE layers where routing is sensitive to tiny changes. Additionally, the post emphasizes the importance of using appropriate metrics, like k3, to measure and manage the divergence, and the need for providing users with granular controls to balance fidelity and performance in their specific workloads.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.