A Review on the Evolvement of Load Balancing Strategy in MoE LLMs: Pitfalls and Lessons

The exploration of Mixture-of-Experts (MoE) architectures has advanced significantly, focusing on load balancing strategies to enhance model efficiency and performance. The evolution began with GShard, which introduced the concept of sparsifying models to manage billions of parameters without a proportional increase in computation, laying the groundwork for subsequent innovations. Key developments include the Switch Transformer’s simplification with single-expert routing, GLaM’s energy-efficient top-2 gating, and DeepSpeed-MoE's focus on both training and inference efficiency through dynamic token redistribution. Recent advancements like ST-MoE and DeepSeek-V3 further refine these concepts by introducing stability measures like router z-loss and bias-based balancing, respectively. These efforts highlight a shift toward more efficient and specialized expert utilization, addressing pitfalls such as over-reliance on auxiliary losses and inference bottlenecks, thus paving the way for more dynamic and adaptive MoE systems. This ongoing progress reflects a broader trend towards integrating high-performance computing strategies and adaptive gating mechanisms to optimize both training and inference processes in large-scale language models.