Most image captioning models produce fluent but generic descriptions, missing the specific attributes, counts, textures, and spatial relations that make an image unique. Multi-stage systems can recover these details through generation, decomposition, verification, and rewriting, but they incur significantly higher inference latency.
In a new paper, researchers propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding using an InfoNCE contrastive loss. This provides dense visual supervision before any text is decoded, without requiring human-written fine-grained captions or pseudo-captions from a multi-stage pipeline.
The authors instantiate SimLoss in two variants: SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats the embedding model as a black-box reward. In experiments, SimLoss FFT achieved the highest precision among single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, nearly matching the F1 score of the multi-stage method while running roughly 20 times faster. SimLoss GRPO attained the strongest recall.
These results suggest that embedding-space supervision can deliver the quality of multi-stage verification at the latency of a single-pass captioner, offering a promising direction for efficient fine-grained image understanding.