Hugging Face researchers have unveiled DataPrep-Bench, the first unified benchmark designed to measure how well large language models (LLMs), agents, and data-centric workflows prepare training data end to end. The benchmark addresses a critical gap: while training data quality fundamentally determines LLM capabilities, no standardized evaluation existed for the data preparation process itself.
DataPrep-Bench evaluates two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training utility of candidate datasets before downstream training. Both are assessed under a shared downstream-grounded protocol across six domains and multiple base models.
For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k. The team releases Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and matches the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains.
For data quality evaluation, scoring functions are measured by Pearson correlation with downstream performance on a shared candidate pool. The researchers introduce the Distributional Alignment Score (DAS), a distribution-based evaluator using Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy. DAS achieves the strongest cross-model correlation in four of six domains and is the only metric exceeding r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators.
DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation, setting a new standard for the field.