MOPD: how frontier labs merge RL experts into one model and how to do it using TRL
Blog post from Hugging Face
Multi-Teacher On-Policy Distillation (MOPD) is a post-training approach that combines specialized reinforcement-learning experts into one general-purpose model by having a shared student learn from a domain-specific teacher for each prompt. The process typically begins with general supervised fine-tuning, creates separate RL-trained experts for domains such as mathematics, coding, instruction following, or agentic tasks, and then distills their token-level feedback into the original shared checkpoint. Unlike standard supervised distillation, MOPD trains on the student’s own generated responses, with teachers scoring those responses token by token, which can provide denser learning signals than a single reward per answer. The technique has been reported in models from organizations including Xiaomi, DeepSeek, NVIDIA, Liquid AI, Moonshot, and Mistral. TRL now supports asynchronous MOPD through AsyncDistillationTrainer, allowing prompts to be routed to expert vLLM servers using teacher identifiers while generation, scoring, and training proceed concurrently. An Open-MOPD example using math, code, and instruction-following experts showed short-run improvements over the starting model, but also demonstrated that long math and code responses can dominate token-weighted training, leaving shorter instruction-following examples underrepresented and limiting gains in that domain.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| AI Model Fine-tuning | 1 | 613 | 130 | 56 | -29% |
| Secrets Management | 1 | 2,500 | 469 | 136 | -19% |
| Serverless | 1 | 691 | 209 | 83 | -50% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.