Large language models (LLMs) have accelerated progress in autonomous agents, but standard evaluations still focus on static task solving. A new frontier is harness evolution—the ability of an agent to autonomously optimize its own operating harness. However, benchmarking this capability has been difficult because existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research.
To address this, researchers at Hugging Face introduce Evo-Bench, the first benchmark specifically designed to evaluate models' intrinsic harness-evolving capabilities across three agent domains: Search, Office, and General. The benchmark uses a novel harness-guided construction framework that leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization.
In extensive evaluations across nine frontier and open-weight models, the top models achieved massive absolute gains of up to 16.6 points, closely approaching state-of-the-art human-engineered baselines. Notably, autonomous evolution outperformed artificial harnesses in General tasks and excelled in Search tasks, but struggled in Office tasks that demand highly specific processing workflows.
The analysis also exposed critical temporal anomalies such as early saturation, while demonstrating that synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models. This suggests that harness evolution could become a key capability for next-generation autonomous agents.