Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI describe Nemotron-pretraining-Indic-v1, a forthcoming open corpus containing approximately 2.8 trillion synthetic training tokens across India’s 22 constitutionally recognized languages, including 2.601 trillion native-script tokens and 224.6 billion romanized tokens for 12 languages. Derived from curated English web, mathematics, code, and question-answer sources, the corpus uses an open GPU-accelerated NeMo Data Designer pipeline for document-level translation, automated quality scoring, targeted repair, and script conversion, with source identifiers retained for traceability. Translations are generally accepted at a quality threshold of 75 out of 100, while workflows vary for languages requiring special handling of long documents or script conversion. A continued-pretraining experiment using 10 billion Indic tokens from a subset of the corpus plus natural Indic text improved five reported Indic knowledge and reasoning benchmarks for Nemotron 3.5 Super VL, although the results do not isolate the synthetic data’s individual contribution. The release is intended to support reproducible model training and allow developers to adapt the pipeline to their own materials, while acknowledging that translated content cannot replace naturally authored Indic text, automated assessments may miss errors, and broader native-speaker validation remains necessary.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.