AI can Autonomously Evolve Pretraining Data Curation

Summary

GAIR/SII/FDU/SJTU (cite Mi et al.; arXiv 2603.14420; Data Darwinism Part II). DataEvolve: observer identifies issues, designer writes a cleaning prompt, cleaner executes on samples, judge scores 1-10 with diagnostics; experience pool + strategy pool carry knowledge across 30 generations per category. Applied to 8 Nemotron-CC academic STEM categories (672B tokens) to produce Darwin-CC (504B, ~25% token drop from targeted cleaning not rewrite). 3B Qwen2.5 trained 500B tokens: Darwin-CC 44.13 avg on 18 benches vs raw 40.17, DCLM 42.42, FineWeb-Edu 36.52, Ultra-FineWeb 36.29; MMLU +18.64, MedQA +13.48 vs raw. Evolved strategies converge on L4-style cleaning (noise/format + domain preservation) not Wikipedia/QA rewrite. Ablation: best vs suboptimal strategy +2.93 avg. Fitness is sample scoring, not full training; limited to 8 academic categories.

Keywords

dataevolve · darwin-cc · data-darwinism · pretraining · nemotron-cc · gair · curation

Topics

pretraining data curation, evolutionary strategies

Research notes

• Primary: arxiv abs 2603.14420 (cs.AI). Code https://github.com/GAIR-NLP/DataEvolve (no SPDX on GitHub). Dataset https://huggingface.co/datasets/GAIR/Darwin-CC (gated; card not readable without access). Discord also linked the HF dataset and GitHub. Substantial pretraining corpus — papers_local intake only; do not write datasets_local here.