Existing benchmarks for autonomous research agents tend to conflate several drivers of improvement at once—training frameworks, hyperparameters, compute budgets, and data—making it hard to attribute why one frontier agent outperforms another to any specific research capability. A new paper on Hugging Face, AutoDataBench, proposes a way to untangle that problem by isolating a single dimension: Data Intelligence.
The authors define data intelligence as an agent's ability to understand, manipulate, and improve the data that shapes model capabilities. The testbed is built on a conceptual framework spanning three complementary capabilities: data diagnosis and repair, data organization, and data construction. These are instantiated through three curated optimization tasks while non-data factors are held fixed.
Evaluation covers tool use, retrieval, and knowledge injection, testing frontier LLMs' ability to improve training data through iterative experimentation under task-specific resource budgets. Beyond raw optimization performance, the paper asks a deeper question: do LLMs actually understand what their data interventions do? The authors compare predictions made before training with observed outcomes to look for evidence of data-effect reasoning beyond trial and error, and explore whether iterative feedback helps models better understand how changes to training data affect performance.
In a final contribution, the team shows that reusing AutoDataBench trajectories for mid-training improves downstream coding performance—positioning the benchmark not only as an evaluation tool but also as a source of high-quality training data. Code and resources are available at the project's GitHub repository.