AI research agents are increasingly expected to run experiments and reason about their results, but measuring whether they genuinely understand how a change in one component affects an outcome has remained difficult. A new benchmark called WhatWorkedBench, published on Hugging Face, aims to fill that gap by testing experimental understanding: the accuracy of an agent's predictions about component changes after a budgeted round of experimentation.
In the benchmark, agents inspect code, choose which measurements to take, and then submit a response surface — a table predicting scores for every configuration of component settings. Reference effects are generated through exhaustive CPU execution, in which each component is changed while the others are held fixed. These effects cover combinations of changes across 36 tasks drawn from 30 data sources and eight workflow types, yielding 1,248 configuration records.
The core evaluation combines 4,206 numerical-control records across all eight families with 108 agent episodes across the original six. With eight new measurements, a pair-effect ridge method selects an optimum on 15 of 22 sources and keeps every effect error within 10% of the score range on three. Fitting a Gaussian process (GP) to the same agent observations improves effect recovery — accuracy relative to true effect magnitude — from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort.
On six completed beat-detection and graph submissions, the same-observation GP lifts family-macro recovery from 0.303 to 0.455. On six workflows with six binary options at 20 new measurements, encoding code equivalences — configurations with identical behavior — raises GP recovery from 0.248 to 0.462.
The authors position WhatWorkedBench as a foundation for research on experimental agents, adaptive experimental design, numerical inference, and the use of program structure.