A new research paper from Hugging Face, titled "What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals," systematically examines whether evaluation artifacts—such as tasks, scorers, and reported metrics—actually support the claims attached to them. The authors argue that these artifacts specify a forward computation but do not necessarily license the claim because the historical evidence and alternative semantics needed to replay it may be unbound.
To formalize this missing claim-replay layer, the paper introduces a framework based on a frozen substrate D, a grounded family F, a claim query q, and the resulting identified set. This framework is then applied to a census of all 124 mechanically eligible Inspect Evals units at a pinned commit, providing a comprehensive audit of their claim-relative inference.
The results are striking: 110 of the 124 units stop before deterministic inference because required historical evidence or semantic grounding is unavailable. For the remaining executable cases, the audit reveals that exact values, winners, complete orders, and pairwise relations separate by claim resolution and by primary versus review family. This means that different claims about the same evaluation can lead to different conclusions.
The audit returns typed stops, instability witnesses, and stable substructure rather than forcing a single evaluator meaning or a binary robust/not-robust label. This approach aims to make the claim-to-evidence layer executable and fail-closed, ensuring that unsupported claims are explicitly flagged.
The paper concludes by highlighting the importance of distinguishing between what a benchmark result actually allows us to conclude and what it merely appears to suggest. By making the claim-to-evidence layer explicit, the authors hope to improve the reliability and transparency of AI evaluations.