September 2025 Summaries

Modular has secured $250 million in its third financing round, led by Thomas Tull's US Innovative Technology Fund, raising its total capital to $380 million and valuing the company at $1.6 billion. The funding aims to bolster Modular's development of a unified compute layer for AI, dubbed a "hypervisor for AI," which addresses the fragmentation caused by diverse hardware and proprietary software stacks. The company's platform, which has been downloaded tens of thousands of times monthly and has accumulated over 24,000 GitHub stars, is designed to maximize efficiency, resilience, and cost reduction in AI infrastructures. With a workforce of over 130 people and a global presence, Modular collaborates with enterprises, cloud providers, and hardware leaders like AMD and NVIDIA to create a more open and innovative AI hardware ecosystem. Its platform includes components like the Kubernetes-native Mammoth and the high-performance GenAI serving framework MAX, which offer significant performance gains and optimizations for AI workloads. Modular's recent funding will enable it to expand its platform's capabilities across cloud and edge hardware, aiming to empower developers with a reliable AI infrastructure.

Modular Platform 25.6 marks a significant advancement in creating a unified compute layer for AI, capable of operating from personal devices to large-scale datacenter GPUs, like NVIDIA Blackwell and AMD MI355X. The release showcases peak performance on these GPUs, with Modular's MAX software delivering industry-leading throughput and latency. It extends developer support across Apple, AMD, and NVIDIA consumer GPUs through the Mojo programming model, which simplifies GPU programming and broadens accessibility beyond enterprise-level users. Enhancements from the previous 25.5 release include improved Mojo and MAX APIs, new pip install support, and expanded model support. The platform addresses the growing demand for AI compute by offering a comprehensive software solution, enhancing portability and performance across various hardware architectures, and providing a streamlined developer experience with improved tools and documentation.

The Modular community has been actively engaging with AI experts and developers through events like the Los Altos Meetup and the September Community Meeting, featuring talks and Q&A sessions on topics such as AI compute democratization, voice AI integration, and optimization of matrix multiplication. Collaborative efforts with Inworld and Oracle Cloud have resulted in significant cost reductions and performance improvements by leveraging NVIDIA and AMD GPUs. The community has also been prolific in producing educational resources and open-source contributions, including a series on optimizing matrix multiplication on NVIDIA Blackwell GPUs and projects like Stargine, a game engine written in Mojo, and "Mojo Miji," an online book on Mojo programming. Community contributions are recognized and encouraged, as evidenced by the acknowledgment of first-time pull request merges.

In this blog post, the development and optimization of a state-of-the-art matrix multiplication (matmul) kernel on NVIDIA's Blackwell GPU architecture are explored, focusing on Cluster Launch Control (CLC) optimization. By employing persistent kernels to reduce overhead costs and introducing a hardware-managed scheduler, the implementation achieves significant performance improvements, reaching 1772 TFLOPs, which surpasses the current state-of-the-art. The post details the persistent kernel's ability to optimize scheduling by controlling block tile coordinates, discusses the pipelining of CLC fetches to overlap tasks and address scheduling overheads, and employs techniques like block swizzling to enhance L2 cache efficiency. The optimization journey, including the use of a circular buffer for Tensor Memory and adapting parameters for production shapes, demonstrates how the kernel can achieve 100.6% of cuBLAS performance for specific matrix shapes. This series illustrates the sophisticated programming techniques required to leverage advanced GPU features for peak performance, with future posts promising further insights into high-performance coding with Mojo.

The text delves into advanced techniques to optimize matrix multiplication performance on NVIDIA Blackwell GPUs, specifically achieving up to 85% of the state-of-the-art (SOTA) performance through leveraging the 2SM technique and pipelining. It explains how Streaming Multiprocessors (SMs) can be grouped to access shared memory, enabling optimizations like Tensor Memory Accelerator (TMA) multicasting. This allows SMs to collaborate on loading tiles into shared memory, reducing redundancy and improving efficiency. The document also discusses the 2xSM Matrix Multiply-Accumulate (MMA), where two SMs coordinate to perform large MMA operations using shared memory inputs, thereby decreasing memory traffic. Further optimizations include pipelining MMA and TMA operations using warp specialization and implementing a circular buffer to increase overlap between computation and memory transfer. This is enhanced by double-buffering the output write-out, which allows simultaneous data transfers, thus achieving significant performance gains. The text concludes by indicating future steps to bridge the remaining performance gap to full SOTA, hinting at using features like cluster launch control for persistent kernels.

In this blog post, the authors delve into optimizing matrix multiplication (matmul) on modern GPUs by leveraging advanced programming techniques and hardware features, such as Tensor Memory Accelerator (TMA), tensor cores, and shared memory swizzling, to achieve significant performance gains. The post outlines the use of loop tiling to reduce global memory access, employing shared memory to cache data blocks, and introducing swizzling to resolve shared memory bank conflicts, thus enhancing computational throughput. The authors explain the implementation of a more efficient matmul kernel using TMA for asynchronous data transfers, tensor cores for matrix-multiply-accumulate operations, and a new memory called Tensor Memory (TMEM) to minimize register pressure. Despite achieving a 58x improvement over a naive kernel, the optimized kernel still lags behind cuBLAS performance, but sets the groundwork for future enhancements through pipelining and overlapping operations. The authors conclude by indicating that further improvements in execution scheduling and algorithm design will be explored in subsequent posts, aiming to close the performance gap with state-of-the-art solutions.