Vision-language models can now write fluent, detailed descriptions of images, but reliably tying those words to the exact pixels they refer to remains an open problem. A new research paper, PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection, tackles this gap by formalizing a task the authors call panoptic grounded captioning.
In this task, a model must describe both foreground objects and background regions while grounding each referring phrase with a pixel-level mask. The authors argue that existing approaches combining dense captioning with pixel-level grounding tend to produce either incomplete descriptions or inaccurate segmentation masks.
The work makes three main contributions. First, it introduces PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets, offering dense captions with near-complete pixel coverage and entity-level image-text alignments for both training and evaluation. The authors also propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly measures textual and mask agreement.
Second, the paper formulates phrase grounding as a selection problem over a phrase-conditioned pool of mask proposals. The resulting model, PANORAMA, conditions a pretrained segmenter on contextualized phrase representations to generate candidate masks, then learns to select the ones matching each phrase. Training this interface jointly with caption generation lets each phrase refer to a single region or to multiple instances.
Third, the authors report that PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks, producing precise entity-level segmentations while maintaining detailed, mask-consistent captions. Code, data, and models are available through the project page and GitHub repository.