Linearizing LLMs with LoLCATs

LoLCATs (Low-rank Linear Conversion via Attention Transfer) is a new approach for quickly creating subquadratic LLMs from existing Transformers, focusing on accelerating models and creating fast models more efficiently. The method involves replacing softmax attentions with linear attentions trained to approximate their softmax counterparts ("attention transfer") and adjusting the model by only adjusting with parameter-efficient finetuning (e.g., low-rank adaptation). LoLCATs allows for state-of-the-art linearized quality, drastically reduces linearizing costs, and scales up to 70B and 405B LLMs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.