Hugging Face researchers have introduced AVA-Encoder, a novel framework for learning agent-native video representations. The approach addresses a key limitation in creative AI: the lack of structured video representations that are both faithful to film content and directly usable by AI agents for reasoning and manipulation.
AVA-Encoder transforms a video into a knowledge graph (KG) representation and then reconstructs it back into video. The KG hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between text descriptions and assets, making them easy for agents to understand, query, and edit.
The framework employs a textual-gradient optimization process, where reconstruction differences drive natural-language update directions. This enables Data-Independent Encoding Policy Pseudo-Training in the outer loop and optional Data-Dependent KG Representation Refinement during test-time inner loop.
In experiments, AVA-Encoder improved by 20.7 percentage points over the strongest external baseline. In a controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy outperformed a human-tuned policy while using 74.3% fewer system-prompt tokens. The team released the full framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.