Efficient and portable mixture-of-experts communication

A portable, open-source Mixture-of-Experts communication library is presented to reduce the latency of routing tokens among distributed GPUs during large-model inference, where experts are sharded across devices and nodes. Built on a minimal set of NVIDIA NVSHMEM primitives, the approach uses GPU-initiated networking through GPUDirect Async, nonblocking remote-memory transfers, and separate send and receive kernels to overlap communication with computation such as attention and shared-expert processing. Its dispatch kernels package and send routed tokens to expert-hosting devices, while combine kernels return expert outputs and compute router-weighted results, using symmetric memory, signaling, and limited synchronization to manage sparse, uneven token traffic. Benchmarks on 128 GPUs across 16 InfiniBand-connected nodes report combined dispatch and combine latency of 902 microseconds with GPU-initiated communication, roughly 10 times faster than dense standard all-to-all operations and 3.6 times faster than a CPU-proxy configuration, while single-node NVLink tests show 2.5 times lower latency than DeepEP. Although it remains about twice as slow as highly hardware-specialized multi-node implementations, the library emphasizes compatibility across NVLink, ConnectX-7, EFA, NVSHMEM versions, and other network environments, and is available through the pplx-kernels GitHub repository.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.