Chasing Speed of Light on TPU v6e

HTDYM performance modeling identified Google’s TPU v6e as a potentially cost-effective platform for serving the 31B-parameter Gemma 4 model, despite the accelerator’s limited 32 GB HBM capacity, lower memory bandwidth, and slower inter-chip connectivity relative to NVIDIA H100 systems. Starting from stock SGLang-JAX performance of roughly 32% model FLOPs utilization (MFU) for 8,192-token prefills, the team used profiling and roofline analysis to find inefficient attention tuning, communication-heavy tensor-parallel AllReduce operations, and unnecessary data movement. Increasing attention query block size substantially reduced repeated KV-cache reads, while custom Pallas kernels implemented collective matmuls that overlapped communication with computation more effectively than XLA’s generated code. The optimization process also uncovered issues involving TPU mesh device ordering, implicit transpose copies, and instruction-memory overlay stalls caused by oversized kernels. Further fusing MLP activation operations, normalization, rotary embeddings, and QKV output handling eliminated additional memory traffic and conversions. These changes doubled prefill throughput from 18,228 to 36,669 tokens per second, reduced time to first token from 449.4 to 223.4 milliseconds, and raised MFU to about 63%, illustrating both the value of specialized accelerators for suitable workloads and the importance of measuring performance gaps against hardware limits.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.