A newly posted Hugging Face paper, Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts, proposes a protocol for quantifying whether explanations actually improve forecasts about planned experiments.
The method relies on paired forecasts that share an intervention, forecaster, and outcome while varying the description, a matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake.
Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, the authors report that the frozen credit decision was inconclusive. The preregistered Tox21 ROC AUC interval-score harm test was unmet (D-M = -.0026, 95 percent interval [-.0174, .0104]), and the OpenML joint formation, point-equivalence, and repeatability rule was also unmet. Matched point-accuracy gains over description remained unconfirmed, with Tox21 and OpenML seed-donor intervals spanning zero.
Under the requested DeepSeek V4 Pro configuration, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66 of 144 cards.
Direct-text Flash delivered all 144 notes without a detectable matched point-accuracy gain, while a researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The authors conclude that the protocol measures predictive credit for research-agent benchmarks and scientific forecasting, but that natural-explanation credit remained unconfirmed at the tested donor resolutions.