August 2023 Summaries

This blog post discusses the release of FlashAttention-2, a new algorithm designed to accelerate attention modules in machine learning frameworks. Building upon its predecessor's success, FlashAttention-2 delivers an astounding 2x speedup, achieved through improved parallelism and work partitioning. The authors showcase how to use FlashAttention-2 on Lambda Cloud and share benchmark results for training GPT-3-style models using NVIDIA A100 and H100 Tensor Core GPUs. The results demonstrate a significant 3x or higher speedup over the baseline implementation, with the H100 80GB SXM5 producing more than 2x Tokens/Sec compared to the A100 80GB SXM4. The authors also explore the scalability of FlashAttention-2 on multiple GPUs and estimate the time to solution for training larger models like GPT3-175B. Overall, this release offers a promising advancement in accelerating attention modules, leading to improved performance and cost savings for machine learning applications.

Lambda Cloud has now introduced on-demand HGX H100 systems with 8x NVIDIA H100 SXM Tensor Core GPU instances, offering more flexibility for users looking to build and fine-tune generative AI models. This new addition provides significantly more compute power, enhanced scalability, high-bandwidth GPU-to-GPU communication, and optimal performance density compared to the previously available 1x H100 PCIe instance. The HGX H100 systems are ideal for larger-scale tasks, with features including 80GB vRAM per GPU, 220 vCPUs, 1.8 TB RAM, and 24.3 TiB NVMe SSD storage, making them suitable for training foundation models and LLMs. Lambda Cloud Clusters also offer access to these GPUs, compute power, high-bandwidth networking, and parallel storage at a lower cost than on-prem hardware infrastructure. The company plans to continue adding more cloud capacity and launching features to make Lambda the best cloud in the world for training AI.