Researchers from Hugging Face have published a paper introducing a causal interpretability framework for modern large-scale diffusion transformers (DiTs). The framework combines attention decomposition with targeted interventions across token spans, heads, and layers to separate prompt-content tokens from structural template tokens.
The study reveals that structural template tokens carry little prompt-specific information at the encoder output. However, they surprisingly emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers.
The authors show that these template tokens acquire semantics indirectly: prompt meaning is first injected into image latents and then read back into the template tokens, rather than being transferred directly from the prompt tokens. This challenges the assumption that tokens encoding semantics at input are the same ones that maintain it during generation.
Based on these findings, the team designed a training-free pruning rule. Heads that attend most strongly to prompt tokens are dispensable; pruning them removes 20% of attention FLOPs with only a 1.4-point drop on the GenEval benchmark.
The work also maps how generative computation is organized across heads and depth, separating semantic routing from visual synthesis and progressing from identity formation to propagation and refinement.