Pretraining a 325M Language Model from Zero on an Inference-Only NPU
Blog post from Hugging Face
VIDRAFT_LAB reports pretraining VKORE-325M-NPU-Base, a 325.6-million-parameter decoder-only language model, entirely from random initialization on a commercial NPU marketed for inference, with backpropagation and weight updates performed on the chip. The model was trained for roughly four days across 76,292 steps and 5 billion byte-level FineWeb-Edu tokens, while an identically initialized B200 GPU reference model used the same data, batch size, architecture, optimizer, and learning-rate schedule. On 1,024 unseen text windows, the NPU model achieved loss of 0.7993 versus 0.7994 for the GPU model, a statistically negligible difference that the authors say makes the models indistinguishable at this evaluation resolution. The project encountered stability problems in earlier long NPU runs, but the authors report resolving them with an undisclosed, patent-pending technique; they release the trained weights, training recipe, and evaluation script but not the full training method. They emphasize that the work is evidence of feasible pretraining on inference-focused hardware rather than a speed benchmark or a claim that the small base model is a capable assistant, and plan to test larger models, longer training budgets, and fine-tuning workloads.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| LLM | 3 | 6,578 | 1,243 | 236 | +9% |
| AI Model Fine-tuning | 2 | 868 | 227 | 77 | +17% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.