November 2024 Summaries

The NVIDIA GH200 Grace Hopper Superchip is an excellent alternative to the NVIDIA H100 SXM Tensor Core GPU for large language model (LLM) inference, delivering superior cost efficiency on single GPU instances. It offers better performance and lower costs compared to the H100 SXM, with a 7.6x increase in throughput and an 8x reduction in cost per token when using Llama 3.1 70B inferencing. The GH200 is particularly beneficial for cost-conscious deployments and faster processing, making it ideal for applications that require quick responses or pipeline throughput. However, users should be aware of some gotchas associated with the novel infrastructure, including compatibility issues with certain libraries and tools, such as PyTorch, which may require compilation for ARM architecture. The NVIDIA GH200 is available on-demand on Lambda's Public Cloud at $3.19 per hour, making it an attractive option for optimizing LLM inference workloads.

The NVIDIA GH200 Grace Hopper Superchip is a powerful and efficient accelerated computing platform available on AWS Lambda On-Demand. It features a 72-core NVIDIA Grace CPU with an NVIDIA H100 Tensor Core GPU, connected by a high-bandwidth NVLink-C2C interconnect, offering up to 900GB/s of total memory bandwidth. This results in faster time-to-first-token (TTFT) for models like Llama3 70B. The Superchip is designed for scientific HPC workloads and offers an optimal solution for simulation-intensive applications in fields like material science and fluid dynamics. It provides a high-performance, cost-effective solution for AI/ML and HPC teams, with the option to scale up to large clusters with up to 720 Grace CPUs and 960GB of H100 GPU memory.