Vision-language models (VLMs) still struggle to reason about space in dynamic environments, where objects move and viewpoints shift. According to a new paper, Spatial-Interactor, these models need to perceive local state transitions caused by motion and viewpoint changes, and then integrate them over long trajectories to maintain an updated spatial state.
Existing spatial training largely focuses on static questions about object attributes and spatial relations, which the authors say provides limited direct supervision for state transitions. By contrast, interaction trajectories naturally link a preceding observation, an action, and a subsequent observation, offering direct supervision for local transitions while complete trajectories reveal dependencies among consecutive transitions.
The proposed framework organizes learning into a three-level curriculum: L1 passive world-state transitions, L2 active self-state transitions, and L3 long-horizon interaction trajectories. To support this, the authors built the Learning from Spatial Interaction dataset (LSI-108K) from simulated and real interaction trajectories, with tasks aligned to each level.
Training uses a two-stage strategy. Supervised fine-tuning (SFT) is applied to L1 and L2 for local transition modeling, followed by On-Policy Distillation (OPD) with privileged self-distillation, where a teacher branch given segment-level transition descriptions supervises the student's on-policy chain-of-thought. This is intended to help the student integrate consecutive transitions over L3 long trajectories.
Experiments across multiple VLMs and spatial benchmarks reportedly show consistent gains in both local transition modeling and long-horizon integration. The authors state that code, dataset resources, and four model checkpoints are available through the project page.