Prime Flash MoE - Faster MoE Kernels optimized for Blackwell

Prime Flash MoE is a set of CUDA kernels optimized for NVIDIA Blackwell GPUs that accelerates Mixture-of-Experts feed-forward layers with SwiGLU activations by reducing intermediate memory traffic, reporting up to 2.4× faster performance than PyTorch grouped GEMM and about 2.3× speedup across 4k–128k tokens. Its fused pipeline combines gathering, gate/up projection, SwiGLU, and down projection while keeping activations on-chip, using tensor-map interleaving to place paired gate and up values in the same CTA and treating the down projection as a split-K operation whose partial outputs are accumulated globally. Because split-K reduction traffic grows with problem size, a separate split pipeline materializes the smaller activation tensor once in memory and performs a conventional down-projection GEMM, making it the default for larger workloads. The implementation uses Blackwell-specific features including Tensor Memory Accelerator transfers, hardware swizzling, tensor-memory accumulators, asynchronous barriers, and hardware-assisted reductions that combine expert routing, split-K accumulation, and output scattering. Both bf16 and MXFP8 variants share the overall architecture, while MXFP8 additionally handles block scaling and quantizes intermediate activations on-chip. Benchmarks on NVIDIA B200 GPUs compare the kernels with per-expert PyTorch loops and grouped GEMM baselines under matched routing and quantization conditions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.