A Roadmap for KV Cache Offloading at Scale

As the demand for KV cache grows due to longer context windows and multi-turn sessions, GPU HBM struggles to scale, prompting a shift towards offloading KV cache in inference engines like sglang and vLLM. This roadmap introduces a three-stage maturity framework for KV caching to enhance scalability and efficiency. Stage 1 involves local offloading to extend KV capacity on the same physical host, leveraging DRAM and NVMe storage, which is effective for smaller deployments but limited by local cache isolation. Stage 2, peer-to-peer sharing, treats a cluster's memory as a shared resource, requiring optimized data transfer and cache-aware routing for improved cache hits and scalability. Stage 3 introduces remote persistent storage to make KV cache durable, enabling reuse across sessions, nodes, and cluster restarts, facilitated by a software-defined architecture that avoids dependency on specialized hardware. These stages offer a path to incrementally enhance infrastructure capabilities, accommodating both small and large-scale deployments with varying workload complexities.