Recent advances in video generation have positioned these models as potential world models. However, a new study from Hugging Face researchers introduces a critical distinction: a true world model must not only generate plausible trajectories but also reproduce the distribution of possible behaviors under the same initial observation and action. This requirement, termed probabilistic alignment, is the focus of the new benchmark PAWBench.
The paper, titled "PAWBench: How Far Are We from Probabilistically Aligned World Modeling?", formalizes probabilistic alignment as a distributional criterion. To evaluate video generators as stochastic samplers, the authors introduce PAWEval, an outcome-level protocol that converts repeated video rollouts into empirical distributions over physical behaviors.
Testing across 50 scenarios and eleven current systems, the study finds that no model consistently matches the reference probabilities while recovering the full range of valid behaviors. This gap underscores a fundamental limitation in current world models.
The researchers also explore whether language prompts, initial noise sampling, or model training can reshape the predictive distribution, but the results suggest that these interventions are insufficient to achieve probabilistic alignment. The work aims to serve as a foundation for future efforts toward more reliable world modeling.