Content Deep Dive

Bring Your Own Expert! Extending MoE Models with New Domain Expertise

Blog post from Hugging Face

Post Details
Company
Date Published
Author
Fabian Klemm, Lars Reimann, Andreas Rabenstein, Henrik Klagges, and Robert Dahlke
Word Count
1,909
Company Posts That Month
60
Language
-
Hacker News Points
-
Post removed?
No
Summary

Researchers at TNG Technology Consulting describe a method for extending mixture-of-experts language models with independently trained, domain-specific experts while keeping the original model weights frozen. Applied to Qwen3.5-35B-A3B-Base, the approach added roughly 125 million trainable parameters per expert for Rust and Haskell programming, with router components learning when to activate the new subnetworks. Rust code-generation performance rose from 0.62 to 0.73 pass@1 and Haskell performance from 0.32 to 0.51, while general benchmark scores remained near the base model’s level. Unlike independently trained LoRA adapters, which degraded or lost specialized capabilities when merged, the added experts occupy separate parameter spaces and can be combined by concatenation; a joint model retained gains in both languages, and brief router-only retraining further improved balance. The authors propose that this could support a modular ecosystem of reusable domain experts, although questions remain about post-training retention, scaling to many experts, inference efficiency, load balancing, and potential interference among larger collections of experts.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
AI Model Fine-tuning 15 613 130 56 -29%
LLM 1 3,355 742 191 -49%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.