October 2022 Summaries

The NVIDIA GeForce RTX 4090 is the newest GPU for gamers, creators, students, and researchers, offering significantly higher training throughput and a more cost-effective option in terms of training throughput/$ compared to its predecessor, the GeForce RTX 3090. The RTX 4090's Training throughput/Watt is comparable to the RTX 3090, despite its high power consumption of 450W. Multi-GPU training scales reasonably well for RTX 4090, with most models getting close to double the training throughput with two GPUs. However, some sub-optimal scaling was observed. The RTX 4090 consistently outperformed the RTX 3090 in multi-GPU tests. Despite its high power consumption and larger size compared to the RTX 3090, the RTX 4090 is a great option for deep learning workloads due to its superior performance and cost-effectiveness.

The AWS Lambda GPU cloud has undergone significant upgrades, tripling the speed of persistent storage and increasing its internet connection bandwidth from 1 Gbps to 10 Gbps. This improvement allows for faster data transfer between instances and the cloud. Additionally, users can now choose the region in which an instance is launched, reducing latency and increasing transfer speeds. The resource usage dashboard has also been updated with instance start and termination times, while a simplified process for copying SSH login information has been added. These upgrades aim to make the platform easier to use and provide better performance for machine learning workloads.

We present an inference benchmark of Stable Diffusion on different GPUs and CPUs to shed light on the questions of what hardware is needed for running this state-of-the-art text-to-image model. The findings show that many consumer-grade GPUs can do a fine job, with the most powerful Ampere GPU (A100) being only 33% faster than the 3080 card when it comes to speed. However, A100 outperforms 3080 in terms of throughput by 2.5x. We also observe that half-precision reduces the time for generating a single output image by about 40% for Ampere GPUs and by 52% for the previous generation RTX8000 GPU. The increase is not linear, and the tensor cores on the GPU are saturated when batch size reaches a certain value. Removing autocast speeds up inference with pytorch at half-precision by ~25%. We verify performance gains both on speed and memory usage side. Our observation is that there are indeed visible differences between single-precision output and half-precision output, especially in early steps.

The NVIDIA H100 Tensor Core GPU offers significant improvements over its predecessor, the A100 SXM GPU, in terms of performance and scalability. The new GPU features fourth-generation Tensor Cores, which deliver 3x throughput on Tensor Core data types, including FP32 and FP64. It also has an increased number of streaming multiprocessors and higher clock frequencies, resulting in a 22% increase in SM count and a 30% increase in clock frequency compared to the A100 GPU. The H100 GPU's new FP8 data type quadruples the computational rates clock for clock per SM of FP16 on A100, and with the help of Transformer Engine, it accelerates AI calculations for transformer-based models such as large language models. The GPU also features updated NVIDIA NVLink and NVIDIA NVSwitch technology, which provide 3x increase in all-reduce throughput across eight GPUs within a single node and a 4.5x increase for 256 GPUs across 32 nodes. This makes it particularly useful for model parallelization and large-scale distributed training. In real-world deep learning applications, the speedup varies by workload, with language models benefiting more than vision-based models. Overall, the H100 GPU is optimized for the largest models, specifically transformer-based, whether for large language, vision, or life sciences applications that involve structured sparsity.