Hugging Face researchers have introduced SemComp-Bench, a new benchmark designed to evaluate whether video generation models can actually complete tasks as intended, rather than merely producing visually convincing clips. The work, detailed in a recent paper, defines a novel task called Semantic Task Completion Video Generation, which shifts the focus from appearance consistency to outcome-oriented success.
Under this formulation, a generated video is considered successful only if it achieves the intended outcome and maintains semantic grounding—the correspondence between the reference image and the generated outcome in terms of high-level, task-relevant semantics. The evaluation does not require a full sequence of intermediate steps or conventional visual similarity to the reference image.
To support systematic assessment, the team built SemComp-Data, a dataset covering six domains. Each instance includes a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized instances.
The benchmark protocol uses a vision-language model (VLM) to answer structured binary questions, reporting two scores: the OA Score for Outcome Achievement and the GR Score for Generation Reliability. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding remains a significant challenge.