GPT-OSS on day 0

Perplexity describes how it achieved immediate support for OpenAI’s GPT-OSS 20B and 120B open-weight models on NVIDIA H200 Hopper GPUs through its in-house ROSE inference engine, choosing FP8 rather than the models’ native MXFP4 quantization because Hopper lacks dedicated FP4 tensor cores. The effort required model-specific changes for grouped-query attention with attention sinks, YaRN positional encoding, biased projections, and sparse mixture-of-experts layers using SwiGLU, Top-K-only softmax routing, and bias-enabled DeepGEMM kernels. Perplexity adapted FlashInfer for sink attention, extended MoE execution while preserving bfloat16 precision for routing accuracy, and optimized GPU kernel launch configurations for the models’ smaller hidden dimensions. Harmony chat formatting, including role, channel, and tool-call metadata, was handled in a frontend formatter and parser so ROSE’s token-based backend could remain unchanged. Benchmarking found that single-node deployments avoid inter-node overhead, with different tensor and expert parallelism choices balancing latency and cost for input processing, while EP4/DP1/TP4 generally performed best for output decoding; using the 20B model to speculatively decode for the 120B model yielded only limited gains because their active parameter counts are relatively similar.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.