Bring Your Own Expert! Extending MoE Models with New Domain Expertise
Blog post from Hugging Face
Researchers at TNG Technology Consulting describe a method for extending mixture-of-experts language models with independently trained, domain-specific experts while keeping the original model weights frozen. Applied to Qwen3.5-35B-A3B-Base, the approach added roughly 125 million trainable parameters per expert for Rust and Haskell programming, with router components learning when to activate the new subnetworks. Rust code-generation performance rose from 0.62 to 0.73 pass@1 and Haskell performance from 0.32 to 0.51, while general benchmark scores remained near the base model’s level. Unlike independently trained LoRA adapters, which degraded or lost specialized capabilities when merged, the added experts occupy separate parameter spaces and can be combined by concatenation; a joint model retained gains in both languages, and brief router-only retraining further improved balance. The authors propose that this could support a modular ecosystem of reusable domain experts, although questions remain about post-training retention, scaling to many experts, inference efficiency, load balancing, and potential interference among larger collections of experts.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| AI Model Fine-tuning | 15 | 613 | 130 | 56 | -29% |
| LLM | 1 | 3,355 | 742 | 191 | -49% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.