Scaling and Optimizing Frontier Model Training

Fireworks has announced a multi-year partnership with Microsoft Azure Foundry to enhance the scalability and optimization of training frontier models, especially focusing on Mixture-of-Experts (MoE) models. This collaboration aims to provide the most extensive range of fine-tunable MoE models available on any platform, overcoming challenges related to memory limitations and cluster orchestration. The initiative introduces advanced training methodologies, including LoRA and full-parameter training, to efficiently handle trillion-parameter models using composable parallelism strategies. These strategies involve FSDP, pipeline, context, and expert parallelism, tailored to each model's requirements. The platform facilitates managed fine-tuning and custom training loops, offering significant improvements in speed and efficiency for reinforcement learning (RL) workloads. Additionally, Fireworks is pushing the boundaries of ultra-long context training and precision computing, aiming to achieve substantial throughput gains while maintaining numerical fidelity. This partnership is set to expand the model catalog and improve GPU topology support, ensuring optimal performance across various cluster configurations.