September 2022 Summaries

PyTorch’s native Fully Sharded Data Parallel (FSDP) strategy, supported in Lightning starting with version 1.7.0 through the `fsdp_native` Trainer strategy, helps train large models despite limited hardware by distributing optimizer states and model weights across multiple GPUs. Unlike standard Distributed Data Parallel, where every GPU stores a full model copy, FSDP shards model parameters across devices and uses efficient communication to reduce per-device memory requirements, providing an approach similar to ZeRO Stage 3 while aiming to retain DDP-level performance. Users can customize the implementation through `DDPFullyShardedNativeStrategy`, and the native PyTorch version introduces streamlined APIs and performance improvements over FairScale’s earlier FSDP implementation, including faster model initialization when CPU offloading is enabled. Because PyTorch classifies native FSDP as beta, Lightning’s corresponding strategy is also subject to interface changes, while FairScale is expected to remain focused on research as broadly adopted functionality is incorporated into PyTorch.

A Lightning App built with NVIDIA Omniverse Replicator enables users to generate synthetic, physically accurate 3D datasets with ground-truth annotations for training AI perception models. It addresses the common challenge of obtaining large, high-quality labeled datasets, which can otherwise require substantial time, expense, specialized staff, and manual work while introducing potential human error. Available through a browser, Lightning Cloud infrastructure backed by AWS, or private clusters for compliance-sensitive use cases, the app is intended to make synthetic data generation more accessible to researchers, engineers, and organizations of varying sizes. Potential applications include training models for defect detection, warehouse logistics, inventory management, and other visual recognition tasks without needing to collect equivalent real-world images.

PyTorch is transitioning from Meta to the newly formed Linux Foundation–hosted PyTorch Foundation, a move intended to increase its independence while giving major partners such as Google and Amazon a broader role, although Meta will continue using and funding it. Research highlights include R3 LIVE++, a LiDAR-inertial-visual system designed to improve real-time 3D mapping; CSL, a dataset of 396,000 Chinese scientific papers for NLP research; and GenLoco, a generalized quadrupedal robot locomotion controller transferable across varied robot designs. Engineering news covers U.S. restrictions on advanced Nvidia and AMD chip sales to China, JCPenney’s AI-powered skin diagnostics partnership with Revieve, and expanding adoption of MLCommons’ MLPerf Inference benchmarks. Open-source releases include Chaquopy for Python Android development, Voltaiq’s EV battery analytics library, and OpenMixup for PyTorch-based visual representation learning, while Lightning AI promotes easier PyTorch training on Apple Silicon through its MPS backend and highlights a medical-imaging project modeling multiple sclerosis lesion progression.

PyTorch 1.12 introduced experimental support for Apple Silicon M1 and M2 processors through its Metal Performance Shaders (MPS) backend, allowing compatible Macs to accelerate model training using their integrated Metal GPUs rather than CPUs. PyTorch Lightning 1.7 simplifies use of this backend with little or no code modification, while benchmarks from PyTorch and the community indicate potentially significant performance gains for workloads such as computer vision and natural-language processing. Users can verify compatibility through their Mac’s system information and must ensure that Python and Conda installations use ARM rather than Intel architecture. Apple Silicon combines CPU, GPU, shared memory, and other components into a tightly integrated system-on-a-chip, and MPS translates PyTorch operations into Metal shader kernels for parallel execution. Support remains experimental because some PyTorch operators are not yet available, though nightly PyTorch releases may provide broader compatibility as development continues.

Lightning AI has created a PyTorch Team led by Mike Ruberry, a longtime PyTorch core technical lead joining as its first member and Chief Engineer. Ruberry’s prior contributions include automated mixed precision, NumPy-aligned APIs such as torch.fft, torch.linalg, and torch.special, and work on making PyTorch programs faster and more composable. The team aims to strengthen the PyTorch–Lightning integration, contribute to PyTorch’s future direction, and improve the framework’s performance, usability, extensibility, and scalability. Lightning positions the move as an extension of its collaboration with Meta and its broader platform for machine-learning workflow templates, while emphasizing PyTorch’s researcher-focused, developer-first community and Lightning’s adoption by an estimated 46% of PyTorch users.