In a new research paper, Hugging Face presents VeriPhy, a system designed to evaluate and refine world models by verifying the physical plausibility of generated videos. The authors argue that visual fluency in generated video does not guarantee physical reliability, and that a single scalar quality score cannot indicate which obligation a clip violates or the exact moment of failure.
VeriPhy operates through a text-only planner that compiles the prompt into typed physical obligations and a statically validated execution plan before any frame is observed. During execution, observations gate and scope only declared calls to frozen low-level experts, such as segmentation and tracking, counting, eleven typed physical measurements, depth, OCR, and audio-event detection. Each action returns a provenance-carrying evidence record, which is either a typed measurement or an explicitly tagged learned state.
Typed resolvers and fixed composition map usable records to a three-valued state: supported, contradicted, or unknown, surfaced as plausible, implausible, or abstain. Every verdict is traceable to the evidence that produced it, ensuring full auditability.
The evaluation is anchored in a 1,500-clip corpus of human-annotated flaw records localizing real generation failures in prompt reference, space, and time. On a 149-clip core with 304 flaw records, VeriPhy accounts for 228, compared to 164 for a published question-decomposition evaluator. While recall alone is similar to prompting the same backbone monolithically (which reaches 222), VeriPhy's key advantage is that each decision retains its evidence record and provenance, making traces auditable one verdict at a time and usable as an interface for writing critic verdicts back into generation.