February 2022 Summaries

Ampere GPUs have improved throughput and throughput-per-dollar compared to pre-Ampere generation GPUs, with significant benefits for language models. The Ampere GPU family offers better performance per dollar than Turing/Volta generation GPUs. However, the lower-end GPUs in the Ampere family may be more cost-effective options when considering budget constraints. Scalability tests showed that some Ampere GPUs perform well with multi-GPU training jobs, while others, such as Geforce cards, experience significant bottlenecks. The recommended GPU choices for Deep Learning depend on specific needs, including multi-node distributed training and model size, with the A100 80GB SXM4 being a top choice for large models and A6000 for mainstream research.

Lambda Labs has created a comprehensive presentation on building and scaling team's deep learning infrastructure, which covers decisions associated with cloud, on-prem, and hybrid infrastructure options. The presentation is based on best practices learned from helping thousands of teams build their machine learning infrastructure. It provides valuable insights for those looking to optimize their team's deep learning setup.