March 2024 Summaries

NVIDIA's GH200 Grace Hopper Superchip platform, now available, represents a significant advancement in accelerated computing and generative AI by combining a powerful GPU with a versatile CPU. Engineered to address the demands of generative AI workloads, its scalable architecture includes the Hopper GPU and Grace CPU connected via the high-speed NVLink-C2C interconnect, which offers 900GB/s of bidirectional bandwidth while reducing power consumption. With up to 624GB of accessible fast memory, the GH200 is designed to excel in complex tasks such as large language models and graph neural networks. Initial benchmarks show the Grace CPU's promising performance, with the GH200 delivering competitive results in tests like the HPCG memory bandwidth and NWChem benchmarks. NVIDIA has also announced the GH200's successor, the Grace Blackwell B200, while platforms like Vast.ai have integrated support for the GH200, offering access to top-tier GPUs without significant hardware investments.

NVIDIA's introduction of Confidential Computing within its Hopper architecture marks a significant advancement in protecting sensitive data and AI models through a hardware-based approach. This feature ensures data-in-use confidentiality and integrity via a trusted execution environment (TEE) on the H200 and H100 Tensor Core GPUs, utilizing a confidential virtual machine (CVM) and a secure, attested boot sequence. The process includes a robust chain of trust, secure communication using AES-GCM encryption, and memory isolation, effectively preventing unauthorized access and modification of data and code. The technology is particularly beneficial for sectors requiring high levels of security, such as healthcare, finance, and the public sector, allowing for secure collaboration and deployment of AI models without modifying application code. Although CC-enabled GPUs may exceed the needs of many use cases, they offer unparalleled security for those prioritizing absolute confidentiality. Platforms like Vast.ai are exploring the use of NVIDIA Confidential Computing in their cloud GPU rental services, providing a promising option for secure, on-demand computing solutions.

Answers.ai has introduced a method to train large language models, such as Llama 2 70B, on limited GPU resources by utilizing fsdp_qlora, an open-source software that leverages QLoRA and FSDP. QLoRA combines quantization with Low-Rank Adaptation to train models larger than available GPU memory, albeit with some limitations related to GPU costs and memory constraints. FSDP, developed by Meta, enables efficient model training by sharding parameters across multiple GPUs, surpassing the prior DDP approach that required fitting the full model on each GPU. The guide outlines steps to rent a 2X RTX 4090 instance on Vast.ai, set up an account, and implement fsdp_qlora for model training, along with optional tools like HQQ for quantization. Troubleshooting advice is offered for instances where GPUs are not recognized, with support available via Vast.ai's chat feature.

Vast.ai has released a product update featuring several enhancements and bug fixes aimed at improving performance, user experience, and reliability of its cloud GPU resources. A significant update includes the beta hosting support for AMD GPUs, allowing users with Radeon or Radeon Instinct GPUs to run the Daemon software, with specific guidance provided in the updated hosting guide. The update also addresses various bugs, such as issues with Docker images, geolocation search results, and instance duration accuracy, while introducing new features like the ability to stop/start offline instances and expanded machine listing support for specific NVIDIA GPUs. Additionally, Vast.ai has reorganized its documentation for better user navigation and introduced a new Discord bot named @Vasti for support, reflecting the company's commitment to ongoing improvement and community engagement.