180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
Blog post from Hugging Face
A developer explored running the 180-billion-parameter Qwen3.8-Flash-Next mixture-of-experts model on a 32 GB Mac, where most model weights cannot fit in memory, and compared existing offloading approaches with Strata, an engine that streams selected experts from storage through a bounded cache. After finding that Transformers’ standard offloading copied entire expert banks and performed poorly, the developer built an expert-streaming method that loads only routed experts, uses a shared slot pool with existing grouped matrix-multiplication kernels, and adds model-specific handling for a large n-gram table and Metal operations. With experts quantized to 3 bits and further optimizations, the model reached about 14 tokens per second for generation, roughly 6% behind Strata, while matching its prompt-processing speed; the initial implementation had been substantially slower. The remaining gap was attributed mainly to host-to-GPU coordination overhead that PyTorch’s Metal backend does not currently expose a straightforward way to eliminate, and the speed was still insufficient for the author’s coding-agent workflow. The experiment’s broader outcome is a proposed generic streaming feature for Transformers and a practical study of expert caching, memory traffic, and CPU–GPU coordination on consumer hardware.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Real-time | 9 | 2,990 | 636 | 190 | -28% |
| MLX | 3 | 11 | 5 | 3 | -80% |
| LLM | 2 | 3,355 | 742 | 191 | -49% |
| Vector Search | 1 | 1,097 | 201 | 87 | -45% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.