A newly posted paper on Hugging Face examines how joint multimodal diffusion transformers handle the relationship between video and companion modalities such as 3D body motion or audio. The authors argue that video is a uniquely rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution, while other modalities encode narrower aspects of the same event.
The paper reports that this richness is mirrored by an asymmetry in cross-modal correspondence. Companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. The authors formalize both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap.
To close this gap, the researchers introduce RecCAR (Reciprocal Cross-modal Attention Regularization), a KL regularizer that treats the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it.
Across joint video-motion and video-audio generation tasks, RecCAR is reported to improve the Human Anatomy score from 0.69 to 0.75 and reduce audio-video desynchronization from 0.804 to 0.752, while also improving overall generation quality.