Researchers at Hugging Face have introduced a novel self-supervised method called Self-Supervised Visual On-Policy Distillation (S2VOPD) that significantly improves the performance of small vision-language models (VLMs) without relying on privileged information such as ground-truth annotations, rewards, or larger teacher models.
Traditional on-policy distillation depends on an asymmetry between teacher and student, often achieved by using a stronger teacher or providing privileged supervision. The new approach inverts this idea: instead of adding information to the teacher, it subtracts information from the student by using strongly augmented views. This creates an effective learning signal for free, matching the benefit of a teacher with access to extra information.
Through systematic exploration of visual augmentations, the team found three key insights: (1) asymmetry is crucial—all four augmentation families improved performance, while symmetric self-distillation degraded it; (2) moderate augmentation strength yields the best results; and (3) the augmentation gap must remain task-consistent, as removing question-relevant evidence can cause large but uninformative discrepancies.
Across six fine-grained perception benchmarks, S2VOPD improved Qwen3.5-4B from 70.7% to 77.4%, surpassing all open-source models compared, including Qwen3-VL at 235B, and even outperforming GPT-5.4. Notably, with the same training data, it recovers 96% of the improvement achieved by methods that use privileged information.
The method is detailed in a paper available on Hugging Face, with a project website at williamium3000.github.io/s2vopd.