Advanced Prompt Caching at Scale

Prompt caching is an optimization technique for inference engines, which involves reusing computed key-value (KV) states across requests to reduce costs and latency. While engines like vLLM, SGLang, and TensorRT-LLM handle caching automatically within a single replica, scaling to multiple replicas presents challenges. A load balancer might distribute requests such that identical prompts are unlikely to hit the same cached replica, degrading the cache hit rate. Solutions include session affinity, which consistently routes a user's session to the same replica, and tiered prompt caching, which organizes caches into shared instruction prefixes (Tier 1) and session-specific prefixes (Tier 2) to enhance reuse. The ideal architecture would feature a shared cache accessible by all replicas, but network latency remains a hurdle. Teams can achieve substantial benefits by focusing on session-affinity routing and structured prompt templates while monitoring cache hit rates and time-to-first-token latency. As the field evolves, advanced architectures may become more common, potentially adopted by significant inference providers like OpenAI and Google.