Scientific discovery begins where known problems end. At that frontier, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on results. Evaluating this ability has proven difficult for two reasons: verifying whether a genuinely new hypothesis holds, and determining whether a system discovered it through exploration or merely recalled related knowledge from pre-training data.
To address this, researchers introduce ExplorationBench, which turns the challenge of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds. Their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks.
The benchmark contains two sandboxes: AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks.
Evaluating 10 AI systems, the researchers find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains.
ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.