September 2026 Summaries

A benchmark of remote versus local KV-cache storage for LLM inference tested vLLM and LMCache across two NVIDIA L4 GPU nodes and a Valkey server connected by 10 Gbps networking, using repeated 10,000-token legal and medical prompts to measure time to first token. For a workload exceeding available host memory, remote Valkey achieved a median cached TTFT of 557 ms, nearly six times faster than cold prefill and substantially faster than local NVMe at 1,521 ms, while DRAM cache misses occurred because the configured 8 GB tier could not hold the roughly 16 GB working set. The analysis attributes Valkey’s performance to parallel network reads reaching approximately 1.2 GB/s, whereas the local disk path achieved about 400 MB/s despite NVMe’s higher theoretical capability. In a two-node test, locally stored NVMe cache provided no reuse when requests reached the other GPU, while Valkey retained essentially identical performance and reused all cache entries across nodes. The findings suggest that DRAM is preferable when the KV working set fits in memory, but a shared remote cache can offer greater capacity, cross-node reuse, and, in this configuration, faster retrieval than local NVMe, although it added roughly 110–150 ms to cache-fill latency. The benchmark materials are publicly available, and the author notes that consistent cross-node cache keys require configuring LMCache with a stable hash algorithm rather than Python’s process-randomized default hash.