A newly posted paper on Hugging Face examines a failure mode in rubric-based reinforcement learning (Rubric-RL), a training approach used when no formal verifier exists. In this setup, a judge evaluates each criterion of a rubric and the verdicts are combined into a reward, typically through a weighted sum.
The authors argue that this additive aggregation is the weak point. Under a sum, criteria can compensate for one another: a policy that misses the single decision that matters can recover the lost points with unsolicited advice. In clinical consultation, they report, such a policy scores higher and answers worse — rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model.
The paper stresses that the medical criteria themselves are not the problem. When the same criteria are grouped so they must hold together, wording unchanged, roughly a third of the loss is recovered; simply shortening answers recovers almost none.
To address this, the authors propose Protocol-level Rubrics (ProRubric), which keeps what the criteria ask for but changes how they are aggregated. A checklist is grouped into a few protocol-level dimensions, and a dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is performed once, offline, and leaves the optimizer unchanged.
According to the paper, ProRubric raises appropriateness by 10.8 points without losing coverage and achieves the best seven-benchmark average at both scales. The authors conclude that reward validity depends not only on what a rubric verifies but on how it aggregates. Code is available on GitHub.