March 2026 Summaries

AI agents rely on system prompts for consistent responses, but the repeated computation of these prompts for every request, especially in high-volume applications, incurs significant costs. Prefix caching is a solution designed to reuse previously computed token sequences in AI models, reducing redundant computation. However, while this method is effective for static contexts like system prompts, it struggles with conversational workloads, where each interaction incrementally alters the token sequence, making reuse difficult. This limitation arises because conversations rarely align with fixed token block sizes used in caching, leading to cascading cache misses as the conversation evolves. Consequently, although prefix caching can significantly reduce costs for shared static contexts, it is less effective for dynamic, long-lived interactions. To manage the complexity of evolving conversational contexts, future infrastructure will need to focus on understanding and efficiently managing these dynamic interactions rather than treating data purely as static sequences.

Allen Helton argues that the current industry focus on tiered KV cache storage for AI/ML applications, such as those involving large language models (LLMs), is misdirected because it overlooks the unique characteristics of these workloads compared to traditional caching systems. Traditional caching is optimized for high transaction rates with small objects, but LLMs often require handling massive gigabyte-sized objects, making network throughput rather than storage tier the primary bottleneck. As a result, optimizing for time to last byte (TTLB) and intelligent prefetching becomes crucial for GPU utilization, as it reduces idle times by having necessary data ready before the GPU is free. The article suggests that while storage tiering is a visible and tractable issue, it is not the ultimate solution, and the real challenge lies in predicting and preloading the needed context efficiently to enhance GPU performance, a problem that Helton's team at Momento aims to address.

The development of internal cache platforms has become crucial for large-scale organizations like Uber and Mercado Libre as they transition from using caches as mere performance optimizations to treating them as essential infrastructure. This shift occurs when independent cache clusters, initially managed by individual teams with varying needs, become difficult to coordinate and manage, leading to opaque systems where failures resemble coordination issues rather than straightforward technical faults. Uber and Mercado Libre addressed these challenges by centralizing operations through control planes that manage provisioning, scaling, and maintenance, and by implementing wrapper SDKs that standardize behavior across teams without requiring application-level changes. This approach allows organizations to maintain operational continuity and stability, particularly during infrastructure shifts or high-stress scenarios, by preventing issues such as connection storms and bandwidth limits that can arise from uncoordinated client reactions. The move towards platforms like Valkey reflects a focus on long-term sustainability rather than short-term performance gains, with the platform taking on the responsibility for the health of the entire cache system, allowing application teams to continue building without being burdened by operational complexities.

As the demand for KV cache grows due to longer context windows and multi-turn sessions, GPU HBM struggles to scale, prompting a shift towards offloading KV cache in inference engines like sglang and vLLM. This roadmap introduces a three-stage maturity framework for KV caching to enhance scalability and efficiency. Stage 1 involves local offloading to extend KV capacity on the same physical host, leveraging DRAM and NVMe storage, which is effective for smaller deployments but limited by local cache isolation. Stage 2, peer-to-peer sharing, treats a cluster's memory as a shared resource, requiring optimized data transfer and cache-aware routing for improved cache hits and scalability. Stage 3 introduces remote persistent storage to make KV cache durable, enabling reuse across sessions, nodes, and cluster restarts, facilitated by a software-defined architecture that avoids dependency on specialized hardware. These stages offer a path to incrementally enhance infrastructure capabilities, accommodating both small and large-scale deployments with varying workload complexities.

GPUs, originally designed for rendering video game graphics, have become crucial yet costly components in AI operations, often leading to inefficient usage due to their architectural and operational constraints. Despite their capabilities for parallel processing, the infrastructure that supports AI relies heavily on maintaining session-specific data, known as the KV cache, on individual GPUs to avoid repeated computations. This sticky session routing leads to some GPUs being overworked while others remain idle, creating inefficiencies across large fleets of GPUs. The challenge mirrors problems seen in other stateful systems like distributed databases and CDNs, but with unique complexities stemming from the large size and high cost of GPU memory. Researchers and infrastructure experts are beginning to focus on resolving these inefficiencies, with solutions likely emerging as these fields converge.

Concurrency tracking is a critical tool for Over-The-Top (OTT) service providers to combat CDN leeching, a form of piracy where unauthorized users access content through legitimate URLs and decryption keys, thereby inflating server loads without contributing to subscription revenues. CDN leeching exploits gaps in security measures like DRM and tokenization, allowing pirates to offer the same quality of service at a lower cost, which can lead to lost subscriptions and increased infrastructure costs. By implementing real-time concurrency tracking services, OTT providers can enforce limits on simultaneous streams per account, detect anomalies, and maintain alignment between license issuance and actual usage, effectively mitigating the impact of piracy. This approach allows for precise control and termination of unauthorized streams without affecting legitimate users, offering a robust defense against the misuse of content distribution networks.

A quiet revolution is emerging at the convergence of high-performance caching systems and large language model inference, as exemplified by Momento's development of a hyperscale cache designed to respond in under 100 microseconds. This innovation addresses the challenge of optimizing GPU utilization during AI inference, where caching plays a crucial role by improving system efficiency and reducing costs. The key lies in managing the KV cache, a computed tensor data structure essential for model processing, which can otherwise lead to redundant computations and load imbalance if not efficiently handled. Momento's approach involves smart routing, intelligent placement, and fast data movement to ensure that precomputed tensors are effectively shared across GPUs, thereby minimizing idle GPU time and reducing time-to-first-token by over 50%. This method leverages traditional distributed systems engineering principles to tackle the underexplored data movement layer in AI infrastructure, offering significant financial and environmental benefits by enhancing GPU utilization.

In a rapidly scaling technological environment, the challenge of managing complex systems often outpaces human cognitive capabilities, necessitating improved tooling and architectural strategies. Companies like Mercado Libre and Nubank exemplify two distinct approaches to handling these challenges: Mercado Libre implemented a governance-focused approach with SDK wrappers and centralized control planes to manage and enforce rules across their extensive caching infrastructure, while Nubank adopted a fundamentally different data model with Datomic to eliminate the instability inherent in mutable systems, thus reducing the need for defensive tooling. Both strategies aim to address the inherent constraint of human cognitive limits in understanding and controlling large-scale systems, with Mercado Libre focusing on enforcement and control, and Nubank reducing complexity by stabilizing the data model. This shared understanding underscores the engineering community's move towards evolving infrastructure around human limitations, as seen with the development of Valkey, which addresses structural failures at the project level and reflects wider industry trends discussed at conferences like Unlocked.