Preference distillation has long operated on two core assumptions: that a student's own failed responses are the most informative negatives, and that rejected responses must come from a model at least as large as the student. A new research paper published on Hugging Face challenges both assumptions, showing that smaller frozen models can generate rejects with less inference compute while producing stronger students.
The study, titled Smaller Models, Better Rejects: Preference Distillation Scaling, tested students ranging from 7B to 72B parameters on code generation and mathematical reasoning tasks. The authors found that rejects from smaller frozen models consistently outperformed self-generated rejects, both before and after sequence-level knowledge distillation.
To explain these results, the researchers derived a finite-horizon utility bound for Direct Preference Optimization (DPO) within a linearized feature model. This bound characterizes favorable reject distributions and motivated three interventions: mixing rejects from smaller and student-scale models, reassigning rejects to other prompts while shuffling code tokens, and selecting candidates with lower likelihood under the reference policy.
The experiments showed that increasing the share of smaller-model rejects improved performance, that shuffled rejects still outperformed length-matched gibberish, and that lower-likelihood selections consistently outperformed higher-likelihood ones for every source. These findings suggest that effective rejects preserve task structure while limiting coupling to the reference policy.
The authors invite discussion on how reject sources might be selected before training and whether similar effects appear in other tasks. The paper is available on Hugging Face and has sparked interest in the research community.