Understanding motion in video remains a core challenge in visual learning, as frame-to-frame changes mix two distinct sources: camera motion and object motion. Decomposing these factors has been difficult due to their tight coupling in natural videos and the lack of separate supervision. Researchers from Hugging Face and academic partners introduce the Structured Dynamics Model (SDM), a self-supervised framework that recovers structured motion representations from frozen features of a pretrained image Vision Transformer (ViT).
SDM explicitly separates the dominant source of temporal change (typically camera motion) from residual dynamics (object motion) through future-feature prediction. Instead of using a single entangled latent or unstructured dense tokens, SDM learns a structured decomposition. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data.
The team evaluates SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision.
These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics. The project page is available at lukasknobel.github.io/projects/StructuredDynamics.