Content Deep Dive

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

Blog post from Hugging Face

Post Details
Company
Date Published
Author
Tarek Ziadé
Word Count
2,347
Company Posts That Month
60
Language
-
Hacker News Points
-
Post removed?
No
Summary

A developer explored running the 180-billion-parameter Qwen3.8-Flash-Next mixture-of-experts model on a 32 GB Mac, where most model weights cannot fit in memory, and compared existing offloading approaches with Strata, an engine that streams selected experts from storage through a bounded cache. After finding that Transformers’ standard offloading copied entire expert banks and performed poorly, the developer built an expert-streaming method that loads only routed experts, uses a shared slot pool with existing grouped matrix-multiplication kernels, and adds model-specific handling for a large n-gram table and Metal operations. With experts quantized to 3 bits and further optimizations, the model reached about 14 tokens per second for generation, roughly 6% behind Strata, while matching its prompt-processing speed; the initial implementation had been substantially slower. The remaining gap was attributed mainly to host-to-GPU coordination overhead that PyTorch’s Metal backend does not currently expose a straightforward way to eliminate, and the speed was still insufficient for the author’s coding-agent workflow. The experiment’s broader outcome is a proposed generic streaming feature for Transformers and a practical study of expert caching, memory traffic, and CPU–GPU coordination on consumer hardware.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Real-time 9 2,990 636 190 -28%
MLX 3 11 5 3 -80%
LLM 2 3,355 742 191 -49%
Vector Search 1 1,097 201 87 -45%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.