Matrix Multiplication on Blackwell: Part 1 - Introduction

This series of blog posts explores the process of writing high-performance GPU kernels on NVIDIA's Blackwell architecture, aiming to achieve performance competitive with NVIDIA's cuBLAS library. The series serves as a reference for optimizing Blackwell GPUs, filling a gap in existing documentation for this new architecture. In Part 1, the series introduces the importance of matrix multiplication (matmul) for large language models (LLMs) and outlines a simple GPU implementation in Mojo. It highlights the role of GPUs in executing data-parallel operations like matmul, which constitutes a significant portion of LLMs' runtime. The series also delves into the evolution of NVIDIA's GPU architecture, from Ampere to Blackwell, explaining how each generation has improved computational performance through features like asynchronous data movement, tensor cores, and new memory architectures. The posts emphasize GPU programming paradigms and the potential cost savings from optimizing matmul performance. Future installments promise to explore hardware-specific optimizations and techniques to maximize performance, ultimately surpassing the current state-of-the-art provided by cuBLAS.