PyTorch Lightning checkpoints write up to 95% faster on Google Cloud Storage Rapid Buckets

PyTorch Lightning has improved its cloud-storage performance through fsspec, the common filesystem interface used across providers such as Google Cloud Storage, AWS, and Azure, meaning the underlying fixes can benefit multiple storage backends. In collaboration with Google Cloud, the project integrated Rapid Buckets through GCSFS, replacing legacy REST-based behavior with persistent bidirectional gRPC streams and requiring users only to change their bucket target. The update changes checkpoint writing from a buffer-copy-heavy write() approach to pipe(), which reads file chunks directly and reportedly reduces checkpoint write times by up to 81% on standard Google Cloud Storage buckets and 95% on Rapid Buckets. It also adds object-storage support for Fully Sharded Data Parallel checkpointing through PyTorch Distributed Checkpoint’s fsspec readers and writers, eliminating the need to stage checkpoints on local disks, while new profiler instrumentation exposes training dataloader initialization and prefetch delays. Joint testing cited a 23% reduction in data-loading time on Rapid Buckets and an additional 70% reduction in checkpoint writing beyond the pipe() improvements, with the relevant features available in GCSFS 2026.8.0 or later and PyTorch Lightning 2.6.5 or later.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.