Nebius and WEKA benchmark distributed KV cache for agentic inference on NVIDIA HGX B300

Nebius and WEKA benchmarked WEKA NeuralMesh with Augmented Memory Grid, a distributed KV-cache tier that pools NVMe storage across an eight-node NVIDIA HGX B300 cluster, to reduce repeated LLM context processing in long-running agentic coding sessions. In an eight-hour replay of real traffic against DeepSeek-V4-Pro, using identical hardware, vLLM configuration, active-session pool, and workload for both tests, the shared cache configuration served 55,694 requests versus 23,015 for GPU HBM alone, achieved a 93% cache-hit rate versus 40%, and reduced errors from 2.32% to 0.64%. It also lowered median time to first token from 5.68 seconds to 1.05 seconds and p95 latency from 23.12 seconds to 3.82 seconds, while producing 3.7 times more output tokens per GPU-hour. The results indicate that the shared tier maintained stable performance as contexts grew toward 900,000 tokens, whereas the HBM-only setup experienced rising latency and timeout rates as its working set exceeded GPU memory capacity.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.