Video generative models have advanced rapidly toward high-fidelity, temporally coherent output, yet they still struggle with prompts that demand specialized knowledge, specific identities, physical consistency, or ordered events. A new research paper, VideoGen-Agent, proposes addressing these gaps not by scaling a single generator, but by training a multimodal agent to orchestrate external tools.
The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the original prompt and intermediate observations to guide its decisions. Training proceeds in two stages: supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined with reinforcement learning. A shared policy is trained on a category-balanced dataset spanning six tasks.
To steer learning, the authors introduce a category-aware hybrid reward that evaluates tool-call validity, task-appropriate tool use, and the quality of the generated video. They also release VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure.
On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, rising from 56.5 to 75.6. Upgrading the generation tools further lifts the score to 86.1 without additional agent training, and human raters preferred the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons.
The authors say the results support learning tool use across video-generation tasks and show that a trained agent can benefit from subsequent advances in generation tools. The project page, dataset, and code are publicly linked.