As AI agents advance quickly, the benchmarks used to evaluate them can saturate, losing their power to separate capabilities or expose remaining failure modes. In scientific domains this problem is sharper still: building and refreshing benchmarks demands substantial time, labor, and domain expertise, so evaluation struggles to keep pace with agent progress.
A new Hugging Face paper, AutoSciBench, asks whether scientific-agent benchmarks can be generated automatically and adapted iteratively as capabilities evolve. The framework represents each task at two levels: a high-level concept specifying the scientific domain, data modality, and required reasoning approach, and a low-level recipe describing how the question, environment, and ground-truth answer are constructed and verified.
Agents attempt each task, producing solver trajectories and judge feedback. AutoSciBench uses that feedback to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories then guides the generation of new concepts, so lessons from earlier tasks inform later benchmark construction.
Starting from existing benchmarks, the authors evaluated AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduced average solver accuracy by 22.4 and 25.5 percentage points relative to human-curated benchmarks in computational biology and materials science, respectively, while generated tasks earned higher average quality ratings across all three domains.
The authors note that initial tasks were often easy for the generating agents to solve, but refinement and accumulated experience produced harder ones. The challenge also extended to other solver models, and the generated tasks scored higher than human-curated benchmarks in the rubric-based evaluation. The work suggests scientific-agent evaluation can adapt as agent capabilities advance.