Researchers have introduced Visual Contrastive Self-Distillation (VCSD), a new on-policy self-distillation (OPSD) method for vision-language models (VLMs) that relies solely on input conditioning, removing the need for external teachers, privileged answers, or auxiliary visual evidence.
Traditional on-policy distillation (OPD) requires a separate teacher model, while existing OPSD methods create asymmetry through privileged answers or visual evidence. VCSD achieves this asymmetry by comparing the same EMA teacher's predictions under two conditions: the original image and a content-erased control, with identical prompt and student-generated response prefix.
The token-wise log-probability difference between these two conditions highlights candidates whose likelihood is specifically increased by the instance-level visual content. This contrast is used to sharpen the teacher's original-image distribution within its plausible support, and the resulting full-distribution target is distilled into the student model.
Evaluated on the ViRL39K dataset across six Qwen3-VL and Qwen3.5 models, VCSD consistently outperforms matched OPSD on every model, with gains ranging from +1.76 to +5.33 points. For example, on Qwen3-VL, the seven-benchmark aggregate improves from 62.27% to 67.04% at 2B, 71.30% to 73.16% at 4B, and 72.51% to 76.26% at 8B. The Qwen3.5-9B model reaches 79.24% average accuracy.
VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost. The project and code are available at https://joliang17.github.io/VisualCSD/.