September 2026 Summaries

Intel Gaudi 3 offers strong AI compute capabilities, but vLLM-Gaudi’s handling of sliding-window attention underutilized the hardware when serving Gemma 4 31B. The authors found that the default system allocated full-context KV cache for all layers despite 50 of Gemma’s 60 attention layers requiring only the most recent 1,024 tokens, and fixing this allocation increased usable KV capacity by 3.5 times for 4,000-token requests. Higher batch sizes then revealed that paged KV cache gathering was a major bottleneck because Gaudi’s Tensor Processor Cores had to collect scattered memory pages before Matrix Multiplication Engines could process attention, producing low memory bandwidth utilization. Rather than fully disabling paging or using an existing contiguous implementation that inefficiently assumed global attention, the team introduced contiguous KV ring buffers for sliding-window layers while retaining shared paged memory for global-attention layers. This eliminated gathers for sliding layers, reduced their attention time by 2.5 times, increased their memory bandwidth utilization to 74.8%, and, at a decode batch size of 56, raised overall decode MBU from 37% to 59% and throughput by 60% over the patched baseline.