Researchers from Hugging Face have introduced ShadowDancer, a novel approach to achieving any-action, frame-level control of interactive video world models. The method addresses a key representational challenge: existing interfaces either encode actions loosely, leaving the model to improvise, or require structured signals that are difficult to acquire across diverse dynamics.
ShadowDancer leverages demonstration videos, which specify dynamics frame by frame, but overcomes the limitation that a video shows dynamics only through one particular appearance—a single "shadow" of the underlying dynamics. The approach introduces two key innovations: shadow pairs and cross-shadow prediction.
Shadow pairs are video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by the Shadow Library. Cross-shadow prediction learns actions by predicting one shadow from the other, discarding what the pairing resamples and preserving what it preserves, yielding a unified dynamics representation that drives a block-causal world model.
Experiments demonstrate improved action transfer and long action rollout over strong latent-action and interactive world model baselines across diverse dynamics families, with an average blinded win rate of 86% in rollout comparisons. The method works across first/third-person gameplay, open worlds, human motion, camera, and robot manipulation—without labels, motion estimators, or fine-tuning.