In the era of foundation models, prompt engineering has become a cornerstone technique for improving language model performance. Now, researchers from Hugging Face ask whether video models—increasingly used as foundation models for visual tasks like reasoning—can similarly benefit from visual prompt engineering.
Their new paper introduces Visual Prompt Engineering (VIPE), a method that automatically modifies the task image to boost model performance. For instance, in a visual physics reasoning task asking where a ball lands after passing obstacles, an abstract sketch can be transformed into a photorealistic version using a simple call to an image editing model.
The study finds that VIPE improves video reasoning performance across a range of tasks. Notably, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling—a finding that underscores the unique potential of visual inputs.
“Just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models,” the authors write.
Example videos and further details are available on the project page at visual-prompt-engineering.github.io.