Researchers have released a new paper examining whether AI agents can carry out open-ended scientific discovery, rather than only solving problems with well-defined metrics. The work introduces Station, an open-world environment in which multiple agents simulate a scientific ecosystem.
To support open-ended work, the authors augmented Station with two mechanisms: a Supervisor and periodic Meta Reflection. These are designed to encourage persistent exploration even when intermediate metrics are absent, helping agents avoid early pivots to easier questions.
The team built open-ended tasks from three recent ICLR oral papers, giving agents each paper's main research question while withholding the original results and disabling web access. They then measured how many of the original findings—split into individual criteria—the agents rediscovered.
According to the paper, Station recovered 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4–20.6% for AI Scientist-v2. The comparison matches cumulative experiment time, though the authors note that token budgets and model teams differ.
Ablation and behavioral analyses indicated that combining the two mechanisms improved research coverage and continuity. On two additional exploratory tasks without oracle papers, some agent discoveries closely matched findings reported by human researchers after the knowledge cutoff date—for example, in subliminal learning, agents found that restricting LoRA training to early layers could restore failed trait transfer.
The authors are releasing the code and full research records for inspection, including failures. They identify scientific judgment as a key remaining challenge, noting that agents still spend substantial effort on questions human researchers find uninteresting.