Improving Decode Throughput on Intel Gaudi 3

Intel Gaudi 3 offers strong AI compute capabilities, but vLLM-Gaudi’s handling of sliding-window attention underutilized the hardware when serving Gemma 4 31B. The authors found that the default system allocated full-context KV cache for all layers despite 50 of Gemma’s 60 attention layers requiring only the most recent 1,024 tokens, and fixing this allocation increased usable KV capacity by 3.5 times for 4,000-token requests. Higher batch sizes then revealed that paged KV cache gathering was a major bottleneck because Gaudi’s Tensor Processor Cores had to collect scattered memory pages before Matrix Multiplication Engines could process attention, producing low memory bandwidth utilization. Rather than fully disabling paging or using an existing contiguous implementation that inefficiently assumed global attention, the team introduced contiguous KV ring buffers for sliding-window layers while retaining shared paged memory for global-attention layers. This eliminated gathers for sliding layers, reduced their attention time by 2.5 times, increased their memory bandwidth utilization to 74.8%, and, at a decode batch size of 56, raised overall decode MBU from 37% to 59% and throughput by 60% over the patched baseline.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.