A new research paper from Hugging Face, titled "Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development," argues that evaluating autonomous agents solely by final scores is insufficient. The authors propose a new framework that uses rule-based metrics to characterize within-run behavior through three dimensions: Solution Framing, Execution, and Feedback Control. They also employ controlled comparisons to assess experience reuse within and across tasks.
The study evaluates seven frontier models across 36 long-horizon tasks. The results indicate that current agents operate more like engineering optimizers than fully autonomous researchers. While they can formulate and implement practical solutions, their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare.
Detailed analysis reveals that similar final scores can hide very different process bottlenecks. Experience reuse can either help or mislead subsequent decisions, and harness design substantially affects performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.
The authors hope this study provides a more fine-grained view of where current research agents succeed, where they fail, and what needs to improve next.