Researchers from Hugging Face have published a paper detailing Visual Attribution Distillation (VAD), a novel algorithm designed to improve multimodal on-policy distillation (OPD). In OPD, a privileged-view teacher supervises student-generated trajectories, but the teacher's next-token corrections often mix visual signals with linguistic priors and teacher-specific effects. VAD tackles the core challenge of estimating which corrections are truly supported by visual evidence.
The VAD algorithm works by evaluating the teacher with and without the relevant visual evidence at each student-generated prefix. The difference in centered log-probabilities yields a signed proxy for the visual evidence direction, indicating how the evidence supports or refutes candidate tokens. VAD then projects the original correction onto this proxy, separating it into an intervention-aligned component and a residual, and reconstructs a student-anchored target from the aligned part.
During training, the reconstructed target serves as the primary supervision signal, while the privileged teacher acts as a weak regularizer. Experiments across six fine-grained visual benchmarks at 4B and 9B scales show that VAD outperforms direct privileged-view distillation and visual-advantage weighting. Token-level and controlled-target analyses confirm that the proxy-aligned component is enriched in task-relevant visual corrections and yields stronger target shifts, particularly when evidence refutes a mistaken answer.
The authors conclude that counterfactual target reconstruction offers an effective alternative to source-mixed supervision, providing a more precise way to distill visual knowledge in multimodal settings.