Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

扩展印度语言预训练：2.8万亿个词元与开源数据生成管道

返回文章列表

[0

[-1

企业

+

]

文章

发布于2026年10月16日

点赞

-

[

Will Jennings

WillJenningsDC

关注

nvidia

Kiran Praveen

kipraveen

Sriharsha Niverty

sniverty

Raviraj Joshi

ravirajoshi

Markus Kliegl

mkliegl-nv

Yev Meyer

nv-3mei

Utkarsh Vaidya

uvaidya

22种语言的原生脚本数据、12种语言的罗马化数据，以及用于翻译、质量评估和修复的可复用GPU加速管道。

作者：AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.

我们非常高兴推出Nemotron-pretraining-Indic-v1，该模型涵盖约2.8万亿个词元，包含22种印度语言。该模型将很快与用于创建它的开源GPU加速数据生成管道一同发布。这些语言是印度宪法中列出的22种语言：覆盖97%印度人口（约12亿公民，根据2011年人口普查数据）的首选语言。由NVIDIA与印度顶尖工程学院IIT Madras及其AI4Bharat实验室联合开发，此次发布为印度语言预训练模型构建者提供了重要的新资源。该管道完整保留了生成方案，包括提示词、大语言模型工作流和质量检查，开发者可据此适配自身素材并生成更多数据。

AI4Bharat是IIT Madras的印度语言研究实验室，专注于构建印度语言的开源数据集和模型。其资源如Sangraha（一个涵盖22种印度语言的开源预训练语料库）为开放印度语言模型研究奠定了基础。此次联合开源发布大幅扩展了社区可获取的开源合成印度语言训练数据。该数据集将英语材料（包括数学、代码和问答内容）广泛翻译至印度语言，为团队提供更多训练素材，用于构建能解释技术概念、解决问题并用这些语言回答问题的模型。每条记录均可追溯至原始来源文档，每个输出均与输入进行核对，确保合成文本始终基于真实信息。

AI4Bharat负责人Mitesh Khapra（2025年《时代》百大AI人物）指出核心问题："印度语言技术落后于英语，因为我们缺乏足够的印度语言数据。"我们通过提供更多数据和开源生成方案来回应这一挑战。

该语料库还考虑了人们的实际书写方式。包含所有22种语言的原生脚本数据和12种语言的罗马化文本，覆盖34种语言-脚本组合。我们的工作流结合文档级翻译、质量评估和定向修复，并包含对公式、表格和代码等内容的保护机制。这些数据与自然撰写的印度语言文本相辅相成，后者对于本地知识和文化背景仍不可或缺。

We're sharing the open generation recipe so the community can not just reproduce the data but take this work further. Teams can use the corpus directly, adapt the prompts and quality checks, or bring their own source documents to the workflow. By opening up both the data and the process, we aim to make large-scale Indic data generation more accessible and enable more developers to contribute to stronger models across Indian languages.

Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text. Our technical report on the model covers the full story. We plan to build on this corpus in future Nemotron models.

What is in the corpus?

Figure 1. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.

The corpus has two groups of language configurations:

• Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.

• Native-script data: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.

Here, native-script describes the writing system: the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. For each of the twelve dual-script languages, romanization covers a subset of the accepted native-script translations.

This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.

From English source documents to Indic training data

To build this corpus, we start with documents from 15 Nemotron-CC-v2 subsets, covering web text, question-answer material, mathematics, code, and other specialized content. We use NeMo Data Designer , NVIDIA's open-source framework for generating synthetic data, to run translation, quality checks, and targeted repair. Throughout the workflow, we retain each source identifier so developers can trace an output back to its original document. Clear provenance like this is part of what makes open data practical to audit and build on. Figure 2 shows the workflow for twelve languages.

Figure 2. Four-stage Indic synthetic-data pipeline for twelve languages, implemented using NeMo Data Designer and Gemma-4-31B-it. We translate English documents, assess translation quality, repair low-scoring outputs, and convert a subset of accepted translations into Latin script. The other ten languages use the separate Indic Translate workflow described below.

The twelve-language workflow

For these twelve languages, we translate documents in full so the model can use context across paragraphs, tables, and equations. We selected Gemma-4-31B-it by comparing models on complete documents, checking translation quality and preservation of technical content. Our prompts distinguish prose to translate from content to preserve, including code, equations, URLs, and formatting.

翻译后的内容如下：

翻译完成后，我们使用相同的模型将每个输出与英文源文本进行评估。受多维质量度量（MQM）启发的评分标准，这是一个用于按类型和严重程度分类翻译错误的开放框架，涵盖含义、流畅性、术语、风格和格式等方面。我们仅保留得分至少为100分中的75分（这是一个评分标准，而非准确率百分比）的输出。对于得分较低的输出，我们利用评估反馈进行修复，然后再次以相同阈值进行评估。这一过程从本地脚本和罗马化数据中恢复了4384亿个词元，这些数据已包含在语料库总数中。通过抽样的人类评审和故意引入错误的测试，帮助识别遗漏内容和残留英文，从而改进我们的提示和检查机制。

被接受的本地脚本翻译还为罗马化数据提供了起点。我们将其中一部分转换为拉丁字母，并将每个结果与本地脚本输入进行比对。这将翻译过程与脚本转换过程分离。在我们最初的对比测试中，这种两步方法相比直接从英文生成罗马化文本，产生的幻觉添加更少，残留英文也更少。

十种语言的工作流程

对于其他十种语言，我们调整了工作流程，以解决在对比测试中观察到的长文档质量和脚本稳定性问题。我们使用Indic Translate进行翻译，这是由Bodhan AI（印度理工学院马德拉斯分校孵化的教育领域人工智能非营利中心）开发的针对英文和22种印度语言的文档级翻译模型。Gemma-4-31B-it随后将每个翻译与英文源文本进行评估。对于八种语言，我们保留了整篇文档的处理方式。对于较长的桑塔利语文档，我们翻译重叠窗口并重建文档；对于曼尼普尔语，我们先翻译为孟加拉语脚本，再转换为本地脚本梅特梅克文。这些工作流程保留得分至少为100分中的75分的输出，并仅生成本地脚本数据，不进行基于反馈的重写。

Indic继续预训练后发生了哪些变化？

Nemotron 3.5 Super VL技术报告将Indic继续预训练（CPT）阶段前后的检查点进行了对比。配置的预算为1000亿个训练词元，其中10%（100亿个词元）分配给合成和自然印度语文本的混合数据。剩余的90%保持了先前训练混合数据的相对比例。

所有五个报告的印度语知识和推理基准测试得分均有所提升：MILU、Indic MMLU-Pro、Indic ARC-Challenge、Indic ARC-Easy和Indic GSM8K。

图3. Nemotron 3.5 Super VL在印度语CPT阶段前后的表现。提升幅度在4.98到9.82个百分点之间。该对比报告展示了结合合成和自然印度语数据的训练阶段的变化，而非单独分析合成语料库的贡献。不同基准测试的语言覆盖范围存在差异；这些是阶段级检查点，而非最终模型版本的对比。

来源：Nemotron 3.5 Super VL技术报告，表2。GSM8K排除了奥里亚语，因为评估超出了服务上下文限制。

这些提升支持了合成与自然印度语CPT方案的结合，同时将其他模型和语言的混合选择留待进一步研究。

The accompanying Indic report analyzes source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data for the twelve-language group. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step for those controlled studies.

Expanded Indic capabilities in Nemotron 3.5 Super VL

Beyond the CPT-stage results, how does the released model compare with the previous generation? Figure 4 compares Nemotron 3.5 Super VL with Nemotron 3 Super across Indic task performance and language quality. These results reflect the released model's full training pipeline, including Indic CPT and post-training.

Nemotron 3.5 Super VL achieves higher scores on five of the six task benchmarks shown, spanning knowledge, reasoning, mathematics, code, and translation. Instruction following on IFEval 1.5 decreased by 2.04 points. It also improves across all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages. Together, these evaluations examine both what the model can do and how well it communicates in Indian languages.

Figure 4. Indic task performance and language quality across Nemotron Super generations. Left: six task benchmarks. Right: preliminary IndiVibe conditional preference scores, not direct head-to-head win rates. Labels show changes in percentage points, except WMT24++, which uses chrF points. See the companion Indic technical report for benchmark and evaluation details.

Start building

Load the corpus

Start with the languages and scripts your model needs, inspect a sample, and review the dataset card and license before training. Configurations use language, script, and region tags: for example, hi-Deva-IN denotes Hindi in Devanagari, while hi-Latn-IN denotes romanized Hindi.

The example below asks for the repository ID from the dataset card, then discovers the available splits and streams a sample.

from
datasets
import
get_dataset_split_names, load_dataset

dataset_id =
input
(
"Repository ID from the dataset card: "
).strip()
config =
"hi-Deva-IN"
splits = get_dataset_split_names(dataset_id, config_name=config)
print
(
"Available splits:"
, splits)

data = load_dataset(
    dataset_id, name=config, split=splits[
0
], streaming=
True
)
print
(
next
(
iter
(data)))

Hugging Face Datasets supports streaming so readers can inspect records without downloading the entire corpus. Once the release is available, pin a dataset revision for reproducible experiments.

For a first training study , compare mixtures at a fixed token budget and record the natural/synthetic and native/romanized shares. Track per-language results alongside English capability retention and language/script quality. When creating held-out splits, account for records derived from the same source across languages and scripts.

To build a fuller recipe, pair the corpus with Nemotron post-training data, such as the Chat & Instruction or Post-Training Blends collections. For India-specific synthetic data, seed NeMo Data Designer with Nemotron-Personas-India , synthetic personas in English and Hindi grounded in real-world distributions for India.

Adapt the generation recipe

开源的 Nemotron Bharat 翻译插件为 NeMo Data Designer 提供翻译和评估提示、可选的基于反馈的重写、文档窗口化以及脚本映射支持。通过其配置参考，您可以选择模型和语言、根据领域定制提示内容并设置重写参数。配套的开发者说明解释了工作流程和工程选择。

从少量自有源文档开始，检查翻译质量及代码、数学公式和格式的保留情况，随后在扩大规模前与母语审阅者协作优化提示内容和校验规则。将生成和语料库接纳视为独立步骤：插件返回候选结果和评分，而您的过滤策略决定哪些记录进入训练语料库。

范围与限制

翻译能扩展源材料的可访问性，但不会替代原本用印度语言撰写的文本，也无法保证涵盖本地知识和文化内容。我们建议将本语料库与适当来源的自然印度语言文本结合，并针对目标使用场景评估混合效果。

质量评估主要依赖自动化，辅以抽样人工审核（不同语言的覆盖范围存在差异）。在十二语言工作流程（图2）中，翻译器同时担任评判者。保留文本仍可能包含错误，十语言工作流程需要更广泛的母语者验证。源ID检查可移除处理重复项，但并非对相似内容进行语义去重。

与语料库同步，IIT Madras 的 AI4Bharat 在 NVIDIA 支持下正在构建 Indic-LLM-Eval-Suite，这是一个持续演进的开源基准测试集，用于评估印度语言模型能力。它帮助模型开发者同时评估任务表现和语言质量。详见配套博客《Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite》。

探索语言配置、测试训练混合方案或将开源方案适配至新领域。分享每种语言的结果和生成流程改进，有助于社区在此基础上进一步发展。

进一步阅读与致谢

配套报告《Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs》和《Nemotron 3.5 Super VL and Lightning 技术报告》提供了语料库统计、生成方法、实验和评估细节。评估是本工作的另一重要组成部分。关于基准测试部分，请参阅 AI4Bharat 与 NVIDIA 联合发布的配套博客《Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite》。

本项目由 NVIDIA、IIT Madras、AI4Bharat 和 Bodhan AI 共同完成。我们同时感谢在流程中使用的模型和工具贡献：Google 的 Gemma、Bodhan AI 的 Indic Translate 以及 NeMo Data Designer。更多 NVIDIA 数据资源请参见 Nemotron 预训练数据集合。

本文提及的模型 4

本文提及的数据集 4

本文提及的集合 2

更多该作者文章

使用 NVIDIA Speech Agent Skills 为您的领域微调 ASR

11

2026年10月13日

一个模型家族，两项金牌成果：为 IOI 和 IMO 微调 Nemotron

24

2026年10月7日

社区

编辑

预览

通过拖拽、粘贴或

点击此处

上传图片、音频和视频。

轻点或粘贴此处上传图片

评论

· 注册或登录以发表评论