CulturaX
- Type
- corpus
- Venue
- Hugging Face / UONLP
- Year
- 2026
- Source
- huggingface
- Access
- restricted
- Language
- 167 languages (English, Russian, Spanish, German, French, Chinese, Italian, Portuguese, Polish, Japanese, and 157 more)
- Added
- 2026-07-17T20:18:03.647506+00:00
- Verified
- 2026-07-17T20:18:03.647506+00:00
Summary
CulturaX is a large multilingual dataset of 6.3 trillion tokens across 167 languages, tailored for LLM development, built by combining mC4 (v3.1.0) with all accessible OSCAR corpora (20.19, 21.09, 22.01, 23.01) and applying a rigorous cleaning and deduplication pipeline (language identification, URL filtering, metric-based cleaning, document refinement, and MinHash fuzzy deduplication). The cleaned dataset is 16TB in parquet format (27TB unpacked), with over half dedicated to non-English languages to boost multilingual modeling.
Keywords
multilingual pretraining corpus mc4 oscar deduplication llm minhash
Topics
NLP / Multilingual Language Modeling
Research notes
- Gated dataset requiring contact info sharing and agreement to non-malicious-use terms. Technical paper: arxiv 2309.09400. Associated KenLM language models released at huggingface.co/uonlp/kenlm.