Direct Preference Optimization (DPO) has become a common route to aligning language models through pairwise comparisons, but it rests on an assumption that every observed preference is trustworthy. A new research paper, PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization, argues that real-world preference data frequently breaks that assumption, producing reversed, weak, or ambiguous labels that can drive harmful policy updates.
The proposed method, Posterior Label Correction DPO (PLC-DPO), introduces a latent state for each preference pair — clean, flipped, or tied — and uses the calibrated policy-reference margin as online evidence to infer posterior-like routing weights. Those weights decide whether training reinforces, reverses, or suppresses a strong directional update.
Rather than simply discarding suspicious examples, PLC-DPO combines forward, reverse-direction, and tie-regularizing losses under those weights. The authors describe the approach as reframing noisy preference learning as actively correcting the direction and strength of supervision, and note that it reuses the log-probabilities DPO already requires, without an auxiliary model or extra supervision. Stability features include EMA calibration, warm-up, and confidence gating.
Across 57 dataset-model-benchmark cells, PLC-DPO reports the best mean win rate against DPO at 60.5%, compared with 55.5% for the next-best method. The paper also presents injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics, which the authors say show the routing remains stable and can distinguish flipped pairs from weakly directional ones.