Video captioning has long focused on generating fluent descriptions of what happens in a clip, but existing models struggle to explicitly tie local visual elements to specific reference images. A new research paper from Hugging Face introduces multi-reference image-grounded video captioning, a task that demands factual descriptions with phrase-level grounding to multiple reference images.
To tackle this, the authors propose RefCaptioner, a two-stage post-training framework. It combines mixed-data supervised fine-tuning (SFT) with a novel reinforcement learning objective called Hierarchical Coverage-Discounted GRPO. This approach jointly improves reference selection, phrase-level binding, distractor rejection, and cross-reference consistency, all while preserving the model's general video-captioning capabilities.
To support training, the team constructed a large corpus containing 20,000 videos and 171,354 reference images. They also introduce MRVBench, a benchmark designed to evaluate caption factuality and multi-reference grounding on both real-world and AI-generated videos.
Experiments show that RefCaptioner achieves the best overall performance among open-source models and remains competitive on standard video captioning benchmarks. Human evaluation confirms that annotators prefer its captions, and that they enable more source-faithful video reconstruction with both open-source and proprietary video generators.