Streaming video understanding is a challenging task for multimodal large language models (MLLMs), which must process continuous visual inputs and respond to queries under strict causality and bounded memory. Traditional approaches compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. However, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide reasoning.
To address this, researchers from Hugging Face introduce LatentStream, a progressive latent working memory framework that shifts the paradigm from store-and-retrieve to retrieve-and-internalize. LatentStream comprises three coordinated components: Query-agnostic Hierarchical Streaming Memory, Hierarchical Latent Memory Evolution, and Progressive Confidence-guided Latent Memory Optimization.
The first component organizes visual history into short-, mid-, and long-term levels under a fixed memory budget using Jenks-guided adaptive consolidation. When a query arrives, the second component equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling iterative retrieval and internalization of historical evidence into a compact, fixed-length latent memory. The third component constructs a hierarchical progression reward from group-wise predictive entropy to refine latent memory tokens and retrieved evidence, encouraging increasingly confident reasoning.
Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks. The code is publicly available at github.com/quhongyu/LatentStream.