May 2026 Summaries

MLSys 2026 highlighted significant advancements in inference across both research and industry, with a focus on agentic engineering, KV cache optimization, and leveraging heterogeneous hardware. The conference featured notable trends such as AI agents writing low-level systems code, which requires rigorous verification and efficient feedback loops, and KV cache becoming a crucial distributed system due to its growing memory demands and complexity. There was also a strong emphasis on the benefits of heterogeneous hardware to optimize inference workloads, as seen in various papers discussing the strategic deployment of resources across different accelerator types. Modular, a sponsor of the conference, showcased its solutions that address these trends by employing its unique stack, which supports efficient agentic development, distributed KV cache management, and hardware-agnostic runtime optimizations. Their work emphasizes holistic optimizations across the entire inference stack, enabling significant performance improvements and adaptability to changing industry requirements.

The text discusses the complexities and solutions involved in building a data layer for LLM routing, emphasizing the differences from traditional HTTP routing due to the stateful nature of inference backends. It highlights the need for a data structure that can handle fast concurrent reads, batched writes, and idempotent event processing to manage cached blocks across numerous pods with microsecond-level query latency. The solution involves using a HostBitmap to efficiently track which pods have cached particular blocks, employing sharding to minimize contention, and using Fibonacci hashing to distribute data evenly. The data layer processes block-level events such as registration and eviction, ensuring resilience to pod churn and maintaining accuracy for routing decisions. To efficiently determine which pods have cached blocks, a binary search over cumulative block hashes is used, allowing for rapid prefix matching. The system also incorporates a two-phase removal process to handle pod lifecycle changes without affecting query performance. This architecture enables fast cache-aware routing, essential for effective decision-making in inference orchestrators.

The article explores the development of a new ecosystem using Mojo, an emerging programming language, through the lens of "Agentic Engineering," a term popularized by Andrej Karpathy. The author describes their experience of building a production-level pastebin service, mobin, solely using Mojo, without relying on Python for backend processes. This endeavor resulted in the creation of ten Mojo libraries and a documentation tool, all accomplished in a few weeks with the aid of AI coding agents. By leveraging AI, the developer significantly reduced the manual coding effort, allowing the focus to shift towards architectural decisions and API design. The piece highlights the potential of Mojo to provide Python-like APIs while delivering systems-level performance, underscoring the efficiency gains possible when integrating AI into the software development process. The author emphasizes the importance of open-source collaboration and consistent project structures as key to rapidly expanding the Mojo ecosystem, drawing parallels to the early growth stages of the Rust language community.

Hippocratic AI is advancing the use of AI in healthcare by developing safety-focused AI health agents that engage in patient conversations, addressing a global healthcare worker shortage. Their Polaris system employs multiple specialized models to ensure interactions are clinically safe, achieving lower error rates than human clinicians and scaling to contact tens of thousands of patients daily. Their collaboration with Modular involves integrating the MAX framework into Hippocratic AI's inference pipelines, optimizing performance with NVIDIA B300 GPUs and enhancing latency management. This partnership allows for flexible, hardware-agnostic deployment of large reasoning models, contributing to efficient patient interactions and maintaining trust in highly regulated industries. Hippocratic AI, which has received $404 million in funding from prominent investors, focuses on non-diagnostic, patient-facing clinical AI agents and avoids using its technology for prescribing or diagnosing, aiming to make healthcare accessible worldwide.

Modular, a company focused on integrating cutting-edge agentic programming tools, has recognized the suitability of the Mojo language for modern AI coding agents due to its efficient syntax and robust type system, which help catch errors early. Despite being a relatively young language with limited training data for large language models (LLMs), Mojo's compatibility with various GPUs, including NVIDIA, AMD, and Apple silicon, makes it a valuable asset. To aid AI agents in producing correct Mojo code, Modular has open-sourced its Mojo code and Python APIs, as well as developed skills that refine AI-generated code. These skills facilitate the translation of existing CUDA and Triton kernels into Mojo, offering performance improvements and broader hardware support. The recent release of Mojo 1.0 beta 1 and the integration of these skills into AI coding agents enable developers to update older projects and translate slow Python functions or domain-specific kernels into Mojo efficiently.

Inkwell, a web application designed to create interactive storybooks in real-time, utilizes Modular Cloud's inference platform to achieve low-latency performance, crucial for generating content on-demand without delays. By tapping into Modular's Gemma 4 31B and Flux2 Dev 32B endpoints, Inkwell efficiently streams story text and images, ensuring a seamless user experience where text appears character-by-character and illustrations materialize as users read. The platform's capability to start image diffusion before text generation completes, enhanced by a 420 ms time to first token (TTFT), enables overlapping processes that reduce perceived wait times. Key optimizations include prefetching potential story paths and caching to minimize delays, with an 85% cache hit rate for user choices. Inkwell's architecture benefits from Modular's efficient API, small runtime footprint, and upcoming server-side intermediate image streaming, which provides a more engaging experience by showing image progression in real-time. Ultimately, the platform's fast response times and support for intermediate states redefine the focus from the model itself to the inference platform's capabilities.

HTTP routing has been a stable field with traditional strategies like round-robin and least-connections effectively balancing traffic across identical backends, but the rise of Large Language Models (LLMs) presents new challenges due to their reliance on GPU pods with unique, stateful characteristics. These models require routers to consider KV cache states, hardware specialization, conversation continuity, and multi-step execution, which are not addressed by traditional stateless routing methods. LLM inference involves maintaining KV caches that drastically affect latency, requiring specialized pods for different phases of processing and ensuring session affinity for conversation continuity. Modular Cloud’s inference framework addresses these complexities through a routing layer that employs a data layer for tracking cache states, a decision layer for routing logic, and an execution layer for multi-step request coordination, using composable plugins to adapt to various deployment patterns without rewriting strategies. This novel approach allows for efficient and effective handling of LLM inference workloads by turning routing decisions into modular, profile-based solutions, paving the way for more flexible and scalable deployments in AI infrastructure.

Modular has released version 26.3, which includes the beta version of Mojo 1.0, marking a significant milestone for the language as it approaches widespread adoption. The Mojo 1.0 beta introduces features like safe closures, conditional conformance to traits, and improvements to variadics, along with the new TileTensor type for enhanced kernel performance. The release also brings video generation capabilities to the MAX platform using Wan 2.2, expanding Modular's offerings beyond text, audio, and image generation. The MAX framework sees enhancements in multi-GPU support, distributed-aware tensor types, and an improved eager interpreter execution path. Additionally, Mojo has launched its dedicated website, mojolang.org, to provide users with comprehensive resources and updates.

April was a significant month for the Modular community, marked by a flurry of innovative projects and expansions. Developers utilized MAX and Mojo to create GPU renderers, FFmpeg bindings, raylib wrappers, and more, enhancing Mojo's capabilities across various domains such as bioinformatics and game development. The community was active at events like AMD AI DevDay, where Modular co-hosted a meetup and showcased advancements such as Gemma 4's launch and Mojo's cross-platform GPU navigation kernels. New Modular offices were inaugurated in Edinburgh and San Francisco, further solidifying its global presence. The month also saw the announcement of several new initiatives, including blog series on software pipelining and structured Mojo kernels, as well as the continued progress toward the stable release of Mojo 1.0. The collective efforts of the community were celebrated, with notable contributions highlighted at the April community meeting and plans for upcoming events in Seoul and beyond.