Sort your prompts: a free speed-up for MoE batch inference

Prompt sorting can improve mixture-of-experts (MoE) inference throughput by grouping requests likely to activate similar experts, reducing the number of expert weights that must be read from GPU memory during decoding. Tests using unmodified vLLM on AMD MI300X hardware found that sorting labeled multi-domain prompts increased throughput by roughly 12–17% for Qwen3-30B-A3B and 7–12% for gpt-oss-120b, while embedding-based clustering of unlabeled UltraChat prompts produced gains of about 5% and up to 4%, respectively. The effect arises because domain-specific inputs, including code, math, prose, and different languages, preferentially route tokens to overlapping expert subsets; fewer distinct experts activated per batch lowers MoE layer latency. The approach requires no model, kernel, or scheduler changes and can use domain labels, router-derived activation fingerprints, or small text embeddings followed by clustering, although benefits are smaller for coarse MoEs, very large batches, prefill-heavy workloads, dense models, and latency-sensitive serving. Reordering may also produce ordinary batch-related nondeterminism in greedy outputs, and practical streaming implementations can retain much of the benefit by sorting requests within bounded windows rather than across an entire workload.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.