Rubric-based reinforcement learning (RL) has emerged as a promising approach for fine-tuning large language models (LLMs) on open-ended tasks. However, a new paper from Hugging Face researchers identifies two critical failure modes that limit its effectiveness: Unexplored Criteria (UC) and Suppressed Criteria (SC).
UC occurs when certain rubric criteria are never satisfied by any rollout, meaning the model receives no optimization signal for those behaviors. Existing methods attempt to address this by injecting rubric information as external guidance during rollout generation, but this creates a train-inference mismatch: the policy is optimized on rollouts produced with guidance that is absent at inference time, leading to error accumulation during autoregressive decoding.
SC, a previously overlooked failure mode, arises when criteria are satisfied by some rollouts but their learning signals are lost because scalar reward aggregation assigns them non-positive aggregate advantages. The paper's analysis reveals that SC is remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample.
To address both UC and SC without introducing train-inference mismatch, the authors propose Criterion-Distilled Policy Optimization (CriPO). For UC, CriPO uses a criterion-injection self-teacher and a localized forward-KL loss to inject missing behaviors. For SC, a counterfactual self-teacher locates criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive, preserving useful patterns.
Experiments on medicine and science benchmarks show that CriPO consistently outperforms standard rubric-based RL, achieving stronger final performance with approximately 2× fewer optimization steps.