Sliding window beats linear attention, but only on short answers

A memory-matched study of attention alternatives finds that sliding-window attention (SWA) with attention sinks is a strong, efficient choice for tasks requiring short generated responses, but hybrid architectures that combine a local window with a recurrent linear state perform substantially better on long-form reasoning. The authors distilled Qwen3-4B-Instruct into SWA and “Bifocal” students under identical 576-token-equivalent memory budgets, with Bifocal allocating memory between a sliding window and an mLSTM-based linear mixer that compresses distant context. SWA led by small margins on short-answer benchmarks, where recent context generally remains in the window, while Bifocal solved about 1.8 times as many math problems involving long answers and produced far more correct responses exceeding 4,000 tokens. The results suggest that stacked sliding-window layers do not reliably preserve distant information despite their theoretical receptive field, whereas recurrent state can carry long-range context directly, although a linear state alone performs poorly without enough exact short-range recall. Benchmarking on Qwen3.5-4B-sized models found Bifocal decoding at roughly the same 195 tokens per second as SWA across prompts up to 128,000 tokens, though with slower prompt processing, leading the authors to recommend SWA as a baseline for short tasks and hybrid designs for workloads requiring long generations.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.