October 2024 Summaries

The NVIDIA H200 Tensor Core GPU is a data center-grade GPU designed for large-scale AI workloads, offering more GPU memory while maintaining a similar compute profile as the widely-used NVIDIA H100 GPU. The H200 is highly anticipated for tasks like training, fine-tuning, and long-duration AI processes, but its performance in inference jobs was tested by Baseten. While the H200 GPUs are good choices for large models, large batch sizes, and long input sequences, they offer minimal performance improvements over H100s outside of these situations, making them less cost-efficient for some inference tasks. The GPU's specs include 76% more VRAM at a 43% higher memory bandwidth than the H100 SXM, but its performance in certain workloads is comparable to or better than that of the H100. Baseten tested the H200 GPUs on an 8xH200 cluster and found them to be well-suited for large models, large batch sizes, and long input sequences, offering significant performance improvements in these areas. However, their performance in shorter context and output workloads is comparable or slightly better than that of the H100. Overall, the H200 GPUs are incredibly powerful and capable GPUs for a wide variety of AI/ML tasks, especially training and fine-tuning, but may not be the best choice for all inference tasks due to their higher cost per hour compared to H100s.

Amazon Web Services (AWS) has launched new on-demand instances of NVIDIA H100 SXM Tensor Core GPUs in its Public Cloud, Lambda. These new instances offer higher-end GPU acceleration for AI developers and provide more affordable options than existing instances. The new 1x, 2x, and 4x instances offer performance gains of up to 49-51% compared to the PCIe version, with features like faster memory bandwidth and higher power draw. The introduction of these new instances creates a solid set of options for AI developers to find the right compute at the right price for their workloads. Additionally, AWS is offering a chance for users to win full-time access to a 96GB VRAM NVIDIA GPU instance for six months by participating in its Golden Ticket promotion.

The Lambda Golden Ticket prize draw offers full-time access to a 96GB VRAM NVIDIA GPU instance for six months, valued at $18,500, to a winner selected from eligible participants in October. Eligibility requires being a direct customer of Lambda with a business email domain and located in the US. Participants can earn tickets by spending a certain amount on NVIDIA H100 Tensor Core instances in the Public Cloud during October. The draw is open to accounts successfully invoiced for their October usage, and bringing teammates can boost chances of winning. To participate, sign up or log in to Lambda's account, build, invite teammates, and start bidding for the Golden Ticket.