How do you make an LLM, anyway? Microsoft just published a textbook.
Blog post from Arize
Microsoft has published a detailed 109-page technical report on their large language model (LLM) MAI-Thinking-1, breaking the industry's recent trend of keeping model training processes secret. This report reveals the extensive process of LLM training, which involves scraping vast amounts of internet data, followed by rigorous filtering to remove unwanted content and deduplication to prevent memorization. The training includes pre-training, mid-training, and post-training phases, with a significant emphasis on using programming code and mathematical texts to enhance reasoning capabilities. Mid-training involves refining and lengthening the document context, while post-training employs reinforcement learning to instill behavioral traits like helpfulness and safety. Microsoft's approach emphasizes small-scale experiments to guide large-scale model training and highlights the avoidance of synthetic data in pre-training, though synthetic tasks are used later to enhance model capabilities. The transparency of this report aligns with Microsoft's strategy to promote their Frontier Tuning service, which offers customized model training for enterprises, assuring them of clean and traceable data usage.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.