Hosting Qwen on Blackwell

Perplexity describes adapting its inference engine to serve post-trained Qwen3 235B mixture-of-experts models on NVIDIA GB200 NVL72 racks, whose 72 Blackwell GPUs combine large HBM capacity, 72-way NVLink connectivity, NVLink Switch SHARP reductions, and InfiniBand links to support high-throughput, lower-cost model deployment. The system separates computationally intensive prompt prefilling from memory-bound token decoding, using tensor and expert parallelism across four GPUs for prefill and data plus expert parallelism, scaled up to 16 GPUs, for decoding; KV caches are transferred flexibly between these differently sharded stages. Custom CuTeDSL kernels and NVLink-based communication improve all-reduce and MoE dispatch-and-combine operations, while SHARP reduces communication overhead and reportedly lowers prefill and decode latency by about 10 percent in practice. Compared with Hopper-based deployments, Blackwell’s greater compute capability, memory bandwidth, and wider NVLink domain enable better scaling beyond eight GPUs, with decoder sharding across 16 ranks justified by faster MoE matrix operations despite modest added communication latency. The deployment uses MXFP8 weight quantization and speculative decoding to improve performance while preserving acceptable accuracy, while more aggressive 4-bit formats remain under consideration because of their accuracy impact without quantization-aware training.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.