Why MoE models get more from speculative decoding

Mixture-of-Experts (MoE) models enhance the efficiency of speculative decoding (SD) in large language models by utilizing a subset of parameters for each token, but challenges arise due to potentially increased weight loading during verification of multiple tokens. The prediction by MoESD indicates a non-monotonic speedup curve where the benefits of SD initially rise with batch size before declining. This study examines temporal correlation in expert routing within MoE models, which reduces the verification cost by decreasing the number of unique experts loaded, particularly at smaller batch sizes. Additionally, at very low batch sizes, fixed-overhead amortization offers a speedup beyond what is explained by routing analysis alone. The findings suggest that optimizing model sparsity and the ratio of shared-to-routed experts can maintain the model in a bandwidth-bound regime, maximizing the benefits of SD at specific target batch sizes. This work highlights the nuanced interplay between sparsity, arithmetic intensity, and batch size in achieving efficient text generation, reinforcing that MoE's sparsity is not merely a complication but a strategic advantage in certain contexts.