July 2025 Summaries

Modular and SF Compute have partnered to revolutionize AI inference economics by launching the Large Scale Inference Batch API, which supports over 20 state-of-the-art models across various domains and promises up to 80% lower costs than traditional methods. This collaboration aims to address inefficiencies in AI infrastructure by leveraging a real-time spot market for GPUs and Modular's high-performance inference stack, thereby offering a more flexible, cost-effective solution for deploying AI at scale. The partnership overcomes traditional constraints imposed by rigid hardware silos and fixed cloud provisioning by enabling seamless allocation across diverse compute backends, effectively redefining AI deployment economics. The initiative not only reduces costs but also aims to eliminate vendor lock-in and artificial scarcity, fostering innovation and expanding compatibility for persistent, low-latency applications.

Modular Inc. has launched two applications, MAX High-Performance GenAI Serving and MAX Code Repo Agent, in the AWS Marketplace under the new AI Agents and Tools category, allowing customers to easily access and deploy AI solutions using their AWS accounts. The MAX platform enhances AI inference deployment with significant performance improvements, optimizes GPU usage across NVIDIA and AMD hardware, and streamlines development workflows, enabling production-grade AI application deployment with minimal code changes. The offerings cater to industries such as financial services, healthcare, and enterprise software development, promising up to 10x faster inference speeds and up to 40% reduced operational costs. Key features include over 500 pre-optimized models, OpenAI-compatible API integration, intelligent code assistance, and repository-aware Q&A capabilities. Modular's approach simplifies the procurement process, centralizing purchasing and control through AWS accounts and offering significant technical innovations like a GPU-native inference engine and cross-platform compatibility.

In June, the Modular ecosystem experienced significant advancements, highlighted by the release of Modular Platform 25.4 and a strategic partnership with AMD, enabling seamless deployment across AMD and NVIDIA hardware without code modifications. The release improved throughput on BF16 workloads and expanded hardware support, while the introduction of Mammoth facilitated scaling GenAI inference across GPUs. The Modular Hack Weekend brought together global developers to innovate in GPU programming, leading to projects like GPU-accelerated quantum simulators and bioinformatics libraries. The community also explored enhancements in Mojo and MAX programming, with increased integration into Python workflows and the launch of the "Mojo Miji" guide for Python enthusiasts. Furthermore, the ecosystem expanded with Modular's availability on AWS Marketplace, facilitating access to pre-optimized models and contributing to rapid AI infrastructure development, as demonstrated by Inworld's new speech pipeline.

Modular Hack Weekend brought together developers globally for a virtual hackathon centered on GPU programming and model implementation using Mojo and MAX, with a focus on rapid development within 48 hours. The event began with a GPU Programming Workshop featuring talks by industry leaders such as Chris Lattner and Bin Bao, which provided foundational knowledge on writing efficient kernels in Mojo and leveraging MAX for model graph construction. Participants collaborated both online and in-person, supported by partners NVIDIA, Lambda, and GPU MODE, which offered resources like GPU prizes and cloud compute credits. Notable projects included Martin Vuyk's GPU implementation of the Fast Fourier Transform, Seth Stadick's Mojo-Lapper for interval overlap detection, and Thomas Trenty's QLabs, a quantum circuit simulator, showcasing significant performance improvements and innovative applications. The hackathon underscored the potential of GPU programming and the supportive community fostered by Modular, with ongoing opportunities for learning and collaboration through their platforms.