Cross-Node Expert Parallelism: DeepSeek's Leap in Throughput and Latency Efficiency

DeepSeek-V3/R1 employs Cross-Node Expert Parallelism (EP) and a prefill-decode disaggregation architecture to enhance inference performance, addressing the challenges of slow inference speeds and rising costs encountered by traditional parallelism methods. By distributing workloads across multiple GPUs and leveraging advanced load balancing techniques, it achieves significant improvements over the standard vLLM framework, with an input throughput of 73.7k tokens per second per H800 node and an output throughput of 14.8k tokens per second during decoding. The system's design principles focus on increasing throughput and reducing latency through communication-computation overlapping and optimizing load distribution across GPUs. Despite the complexity added by EP, this approach effectively maximizes resource utilization, minimizing latency bottlenecks and ensuring superior performance. The service, primarily run on H800 GPUs, reports a peak node occupancy of 278 and generates substantial theoretical revenue, although actual earnings are lower due to factors like pricing differences between DeepSeek-V3 and R1 and free access for certain services.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.