Hugging Face researchers have released CAPEval (Coverage And Precision Evaluation), a new benchmark designed to decouple caption quality into two distinct dimensions: Coverage and Precision. The work addresses a fundamental flaw in existing evaluation methods that treat caption quality as a single scalar score, conflating how much visual information a caption covers with how reliably the image supports its claims.
CAPEval decomposes caption quality into Coverage, which quantifies how thoroughly a caption covers ground-truth factual content, and Precision, which reflects the factual correctness rate of all claims expressed in the caption. The benchmark includes human-written ground-truth captions and human-verified atomic checklist items to ensure reliable evaluation.
In their experiments, the team selected 10 captioners and conducted controlled downstream end-to-end experiments across four model families, where the caption source was the only variable. The results reveal a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, while Precision acts as the dominant predictor for generation performance.
This decoupled evaluation paradigm offers a more fine-grained diagnosis of caption quality and provides actionable guidance for selecting and optimizing captioners tailored to different downstream tasks. The code is available on GitHub at https://github.com/liuzhipenggg/CAPEval.