Robot foundation models can perform well on data resembling their training distribution but often falter when the visual world changes. According to a new paper, part of the problem is a vision-action shortcut: when models learn to generate actions from pretrained visual representations, they may latch onto visual cues that merely correlate with demonstrated actions in the training data. When those correlations shift, performance drops.
To address this, researchers propose Latent Interface Training (LIT), a framework-agnostic, two-stage strategy. In the first stage, an action expert is trained to generate action chunks conditioned on language, robot state, and the terminal SE(3) end-effector pose of each demonstrated chunk — with no images involved. This establishes a goal-directed action prior that is independent of visual cues.
The second stage adds a latent interface that aggregates visual and semantic representations and becomes the action expert's only visual conditioning pathway. This interface is supervised to reconstruct the terminal pose used in Stage 1, pushing it to retain the goal-relevant spatial information needed for action generation while limiting task-irrelevant visual influence.
The authors tested LIT across four vision-language-action and world-action architectures: Pi0.5, MolmoAct2, FAST-WAM, and ImageWAM. On LIBERO-Plus, LIT improved overall success by 3.87 to 10.70 percentage points while preserving or improving average LIBERO success. In real-world evaluations spanning three tasks, the method delivered 13.30 to 16.70 percentage-point gains under unseen camera configurations, lighting variations, and distractors.