Inside the Nebius + PyTorch DeepSeek V3 recipe: NVSHMEM and DeepEP for wide expert parallelism

In 2026, nearly all frontier models in machine learning are mixture-of-experts (MoE), which rely heavily on expert parallelism (EP) to efficiently distribute model training across multiple GPUs. However, this approach encounters significant communication bottlenecks due to the all-to-all data exchanges required in each layer, making traditional NCCL collectives inefficient. The document discusses GPU-initiated RDMA using NVSHMEM and DeepEP as a solution to this bottleneck, enabling direct GPU memory access and reducing latency in communication. By implementing these technologies, particularly on platforms like Nebius, significant improvements in model FLOPs utilization (MFU) and training throughput have been observed without increasing memory costs. The text provides insights into the challenges of MoE models, the advantages of RDMA over traditional collective communication, and the specific tools and configurations necessary to optimize performance in large-scale distributed machine learning tasks.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.