Researchers have unveiled ASPIRE, a novel benchmark designed to test whether large language models (LLMs) can self-evolve from vague, high-level goals such as "become a better physicist" or "improve at research." Unlike traditional self-evolution setups that rely on human-specified tasks and metrics, ASPIRE provides only a natural-language capability goal while keeping downstream evaluation tasks hidden.
The benchmark requires agents to operationalize the goal by selecting their own data and update methods, constructing training and validation signals, and deciding when to evaluate. It supports both model-weight and agent-harness evolution in a unified interactive environment, with hidden expert-authored evaluation sets spanning 520 items across six goals.
Experiments reveal that vague goals shift the focus toward interpreting the goal itself. While current agents can complete training and harness-editing loops, weight-level improvements remain sparse and unstable. The strongest evolved harness still underperforms the engineered Qwen-Agent reference, and agents often train on mismatched data or rely on narrow self-evaluations, causing local gains to fail on hidden tests and later training to erase earlier progress.
The findings highlight a critical gap: models can execute optimization loops but struggle to decide what and how to learn without explicit guidance. The project page is available at self-developing-agents.github.io.