CuTeDSL at Perplexity

Perplexity’s in-house Runtime-Optimized Serving Engine, or ROSE, serves embedding models, LLMs, ranking systems, and other transformer workloads across NVIDIA Hopper and Blackwell GPUs, emphasizing rapid adaptation to new model architectures alongside high inference performance. Although ROSE initially relied on Triton and uses CUTLASS and cuBLAS for matrix multiplication and attention, it has adopted NVIDIA’s Python-based CuTeDSL as its main environment for many specialized GPU kernels because its just-in-time compilation, low-level hardware access, MLIR-to-PTX pipeline, and layout abstractions enable aggressive specialization without CUDA’s lengthy template-compilation burden. The system differentiates prefill workloads, which prioritize compute throughput over many tokens, from decode workloads, which are typically memory-bandwidth and latency constrained, adapting warp counts, grid assignments, vector widths, and synchronization strategies accordingly. CuTeDSL has supported optimized QK normalization kernels that reportedly improve throughput and latency by two to three times, as well as mixture-of-experts dispatch and combine kernels for local, tensor-parallel, NVLink data-parallel, and InfiniBand deployments, including a reported twofold improvement over previous CUDA kernels for NVLink token transfer and a further latency gain for peer-to-peer kernels. The DSL also eases Blackwell support through configurable SM counts, wider memory instructions, and newer quantization formats, while internally developed CuTeDSL code patterns have gradually made AI-assisted kernel generation more useful despite models’ continuing difficulty with low-level synchronization and vectorization. Perplexity concludes that CuTeDSL approaches CUTLASS C++ performance while improving development speed and configurability, positioning ROSE to further fuse computation, communication, normalization, and activation operations across its inference workloads.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.