Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUs

Google’s MaxText team reproduced AI2’s open OLMo 3 7B language model on Cloud TPUs using JAX/XLA, matching both its large-scale stage-1 pretraining run of roughly 5.93 trillion tokens and its stage-2 mid-training anneal against AI2’s PyTorch/GPU references. They ported OLMo’s specialized architecture, converted initial checkpoints with close logit parity, and validated fidelity through held-out C4 loss, multi-domain perplexity, token-level KL, and an eight-task benchmark suite rather than relying only on training loss. This evaluation exposed a double-sharding data-loader bug that made training loss appear better through memorization while held-out performance remained unchanged, underscoring the importance of generalization metrics and exact checkpoint-resume testing. The project also demonstrated that the training recipe could survive preemptions, reproduce losses exactly after resume, scale efficiently between TPU slice sizes, and move from Ironwood to v5p TPUs without altering the core launcher. Performance optimizations achieved 44.5% model FLOPs utilization on Ironwood and 57.4% on v5p, while a separate attention-head reshaping experiment increased throughput by 12.4% at unchanged parameter count and FLOPs. The authors conclude that cross-framework reproductions can be faithful within run-to-run variation when data handling, optimizer precision, deterministic resumption, and held-out evaluation are rigorously controlled, with long-context adaptation and post-training work planned next.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.