Autoregressive video diffusion models generate long videos by producing chunks from a bounded recent context. While this recency-based caching preserves local continuity, it often discards historical cues needed when subjects, objects, scenes, or attributes reappear later. Existing memory mechanisms expose models to nonlocal history, but access alone does not ensure effective use.
In a new paper, Hugging Face researchers analyze video diffusion transformers (DiTs) and find that different layers exhibit distinct preferences for current, recent, and distant context. This suggests that long-range memory requires deciding both what to retrieve and where to use it. To address this, they propose LayerRecall, a current-conditioned, layer-selective memory router that retrieves relevant historical key/value states and injects them only into backbone-specific memory-sensitive layers, while preserving local attention elsewhere.
To reduce reliance on scarce high-quality long-horizon videos and explicit memory-allocation labels, the team also introduces Cross-Horizon Prediction Matching (CHPM). CHPM uses a privileged long-context reference to supervise the bounded-memory router in prediction space, enabling training without manual annotations.
Across 100 multi-shot evaluation prompts, LayerRecall achieves the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long, demonstrating stronger long-range recovery without sacrificing local continuity. Qualitative analyses reveal memory-guided self-correction, where initially mismatched local attributes return to their historical appearance without resetting ongoing motion or scene structure. Additional analyses show cross-backbone portability and negligible inference overhead.