FP8 KV-Cache on Intel® Arc™ Pro B70: 2× Capacity with strong Long Context Throughput Gains

FP8 KV-cache quantization in vLLM was evaluated on Intel Arc Pro B70 GPUs across 10 language models ranging from 1B to 72B parameters and tensor-parallel configurations of one to four cards, showing a deterministic doubling of KV-cache token capacity because FP8 stores cache elements in half the space of BF16. This capacity gain generally produced roughly twice the raw number of simultaneous 4K-context sessions, while long-context workloads at 16K to 32K often achieved substantial throughput improvements, including a peak 42.3% gain for Qwen2.5-14B-Instruct. Accuracy testing with RULER found near-lossless results for eight models, particularly dense GQA architectures such as Llama, Qwen, Mistral, and a 70B DeepSeek variant, but identified major degradation for DeepSeek-R1-Distill-Qwen-7B and meaningful declines for Gemma-3-1B-IT. The study also found that increased cache capacity rarely improved SLA-qualified short-context concurrency because prefill latency and compute, rather than KV-cache residency, were the main bottlenecks. It recommends FP8 KV as a simple, model-validated option for memory-constrained and long-context inference, while positioning TurboQuant as a more complex alternative for deployments requiring greater compression and concurrency scaling.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.