FineFineWeb: A Comprehensive Study on Fine-Grained Domain Web Corpus

Ge Zhang, Xinrun Du, Zhimiao Yu, Zili Wang, Zekun Wang, Shuyue Guo, Tianyu Zheng, Kang Zhu, Jerry Liu, Shawn Yue, Binbin Liu, Zhongyuan Peng, Yifan Yao, Jack Yang, Ziming Li, Bingni Zhang, Minghao Liu, Tianyu Liu, Yang Gao, Wenhu Chen, Xiaohuan Zhou, Qian Liu, Taifeng Wang, Wenhao Huang

Open source

Type: corpus
Venue: Multimodal Art Projection (M-A-P)
Year: 2026
Source: huggingface
Access: free
Language: English
Added: 2026-07-17T20:18:03.778319+00:00
Verified: 2026-07-17T20:18:03.778319+00:00

Summary

FineFineWeb is a large-scale web corpus that classifies FineWeb data into fine-grained domains (e.g., aerospace, etc.) using a multi-iteration pipeline involving GPT-4 URL labeling, FastText coarse recall, and BERT fine recall. It contains ~4.89B rows across multiple domains with billions of tokens each, enabling researchers to build domain-specific pretraining mixtures with explicit control over domain composition.

Keywords

web-corpus domain-classification pretraining fineweb curation gpt4-labeled

Topics

Web text / General

Research notes

Published December 2024 (v0.1.0). ArXiv paper listed as 'coming soon' at time of research. Uses 3 iterations of coarse/fine recall with FastText and BERT classifiers.