Optimizing LLM Serving Efficiency: Moving Beyond KV Cache Reuse to Token-Load Awareness with Ray Serve LLM

Large language model serving requires routing requests across replicas while accounting for stateful KV caches, highly variable input and output lengths, and unpredictable generation costs, making conventional microservice load balancing insufficient. Ray Serve LLM offers session affinity, prefix affinity, and KV cache affinity approaches to reuse cached computation, but the post argues that maximizing cache reuse alone can create request herding and uneven workloads when long or high-token requests concentrate on the same replica. Its KVAwareRouter combines actual KV cache overlap, informed by vLLM cache events and NVIDIA Dynamo’s KV indexer, with token-load estimates that capture uncached prefill work and active decode demands. Tests using asynchronous reinforcement-learning rollouts and reconstructed Claude Code agent traces found that this approach may sacrifice some cache-hit rate but improves balance, time to first token, time per output token, throughput, and tail latency for heterogeneous workloads. Consistent-hash session affinity remains useful when cache locality is especially valuable and sessions have comparable workloads, while future work aims to address unknown output lengths and expand token-aware routing to disaggregated, data-parallel, and multimodal deployments.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.