Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. But their reliability depends on whether they reward honest answers over adversarial answers optimized to exploit them — a property that has remained poorly understood.
A new research paper introduces ImpossibleRubrics, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, alongside 48 answerable controls. Rather than providing fixed rubrics, the benchmark supplies task environments and certificates, allowing rubrics to be generated downstream and then adversarially tested for whether they reward certificate-violating answers.
Across eleven rubric generators, exploitation rates ranged from 8% to 26% on an unbiased 150-of-169 environment cut. On a deliberately selected stress cut, the strongest generator measured was still exploited 36% of the time, while a certificate-faithful rubric was exploited 0% — indicating that what is being measured is a rubric-quality gap, not task impossibility.
One finding runs against intuition: a single generic rubric instructing models to "be decisive, penalize hedging" and used unchanged for every task was exploited 64% of the time, and seven of the eleven generators were exploited more often than that while writing a rubric tailored to each task. The tailored criteria appear to tell an attacker which claim to fabricate.
The authors conclude that the problem is not that rubrics are vague, but that they are specific about the wrong things. The benchmark is fully open-source, including the dataset, evaluation pipeline, and baseline results.