Announcing Long-Context Training on TPUs with MaxText Context Parallelism

Google Cloud has added long-context training support to its open-source JAX framework MaxText v0.2.4, enabling large language models to train on sequences reaching millions of tokens across Cloud TPU 5p and TPU 7 systems. The release addresses the quadratic compute cost of attention and the memory burden of storing key-value activations through four context-parallel approaches: all-gather, all-to-all Ulysses, peer-to-peer ring attention, and a hybrid Unified Sequence Parallelism method that combines all-to-all and ring communication. All strategies use the Tokamax Splash Attention kernel, while context parallelism distributes a sequence across chips and can overlap communication with computation to reduce memory pressure. Benchmarks using Llama 3.1 8B reported up to 70% model FLOPs utilization at sequence lengths of up to four million tokens on TPU 5p, while TPU 7 supported contexts up to 14 million tokens and achieved 48.7% utilization at that scale. The announcement recommends all-gather when full key-value data fits in memory, Ulysses for limited chip counts compatible with available attention heads, and ring-based methods for larger contexts, with the hybrid approach often improving scaling where head configurations permit. Future plans include support for models mixing local and global attention, linear attention mechanisms, and further kernel and communication optimizations.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.