A new research paper from Hugging Face investigates the mechanisms behind semantic leakage in audio-video diffusion models. The authors introduce the concept of an "attention triangle," which comprises the three cross-attention edges connecting text, audio, and video streams. They analyze how semantic information is routed across these modalities during generation.
The study reveals that the audio-video edge is bidirectional: audio can influence video generation, and video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage. When prompts conflict with learned priors, cross-modal interactions may override intended conditioning, rerouting semantics toward visually canonical but incorrect outcomes.
The researchers extract attention-derived signals that expose how semantics are distributed across modalities, using them as a diagnostic tool to analyze and deliberately incur leakage under controlled conditions. This allows them to probe internal dynamics and isolate the role of individual interactions.
Building on these insights, they propose inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments demonstrate improved semantic grounding while preserving generation quality. The project page is available at https://sagipolaczek.github.io/The-Attention-Triangle/.