Researchers behind a new paper argue that one of the hardest skills in science is recognizing which buried prior idea a fresh problem actually needs — and that AI systems remain far behind human scientists at it. To study this ability, they introduce ScholarCatalyst, a benchmark for retrieving what they call "catalyst papers": earlier work that did, or could have, advanced a research project.
The labels come from scientists themselves. Using an automated pipeline designed to make author annotation scalable, the team had 184 lead authors of 207 recent computer science papers label which candidate works did or could have helped their project, each with a detailed rationale. The result is roughly 1,000 queries with author-labeled hard negatives.
The benchmark defines a retrieval task: given an initial research question, a system must surface these papers using only the literature that existed when the project began. According to the paper, agentic search performs no better than plain embedding retrieval — 0.42 versus 0.48 Recall@20 — even though the agent calls that same retriever as a tool. An agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20.
The authors conclude that current retrievers and LLMs remain far from matching author judgments, and that new training recipes are needed to give models an expert-like intuition for searching broad corpora. They frame ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
The dataset, code, and a demo are publicly released alongside the paper.