Announcing New vLLM and SGLang Inference Engines on TPU

Google has launched native PyTorch TPU backends for vLLM and SGLang, developed with Inferact and RadixArk, respectively, to let developers use familiar PyTorch tools for deploying and optimizing AI inference workloads on TPUs. Built on TorchTPU, the backends treat TPUs as native PyTorch devices, support eager execution and torch.compile with XLA, retain each framework’s existing serving architecture, and use TPU-specific Pallas kernels for performance. They target production needs including prefix caching, speculative decoding, KV-cache offloading and transfer through TPU Sync, disaggregated prefill and decode, multimodal models, long-context processing, and multiple forms of model parallelism and quantization. Google says both engines are already being used by preview customers and reports public benchmark results indicating low token costs and strong performance on agentic workloads using Ironwood TPUs, while partners cite deployments and model-serving results. The projects support several major open-model families, plan expanded model and performance support, and will coexist with the maintained vLLM-JAX and SGLang-JAX runtimes; vLLM is also introducing a TPU compute pool for open-source contributors.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.