Hugging Face researchers have released PRM-as-a-Judge 1.5, an upgraded toolkit designed to assess embodied robotic models with greater nuance than traditional binary success rates. The new version transforms rollout videos into dense progress curves, enabling a deeper understanding of model behavior throughout task execution.
Building on version 1.0, the toolkit introduces three new metrics that characterize failure-side progress, post-drawdown recovery, and success-side execution quality. These metrics aim to provide a more comprehensive picture of an embodied model's capabilities, moving beyond simple pass/fail outcomes.
The researchers also present RoboPulse++, a tool for evaluating the reliability of process reward models (PRMs), offering a more accurate testing platform for these evaluators. In addition, they release a user-friendly assessment suite that includes benchmark implementations, metric calculations, and visualization tools to support reproducible manipulation process evaluation.
The authors call on the community to rethink robot evaluation, advocating for transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence. The full paper and toolkit are available on Hugging Face.