Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

NVIDIA NeMo AutoModel, an open library within the NVIDIA NeMo framework, significantly enhances the efficiency of fine-tuning Mixture-of-Experts (MoE) models by integrating seamlessly with HuggingFace Transformers v5. It introduces Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, resulting in a 3.4-3.7x increase in training throughput and a reduction of GPU memory usage by 29-32% compared to native Transformers v5. The integration is designed to maintain API compatibility with HuggingFace, requiring only a single import line change to leverage these improvements. This setup allows for scalable training across multiple GPUs, making it feasible to fine-tune large models like the 550B-parameter Nemotron 3 Ultra across 16 nodes. NeMo AutoModel's optimizations include sharding expert weights across GPUs and fusing communication with computation to enhance speed and efficiency, all while maintaining compatibility with standard HF-format checkpoints for easy deployment on various inference frameworks.