Direct preference alignment has become a standard way to steer large language models toward human preferences, largely because it is computationally and memory efficient. Yet the paper argues that likelihood displacement — where preference pairs carry only small likelihood margins — motivates alternative ways of extracting information from those pairs.
To address this, the authors propose Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method built on comparison oracles. Rather than directly optimizing a differentiable preference loss on preference pairs, ComPO extracts directional information from them.
The paper establishes a convergence guarantee for ComPO's basic offline scheme under assumptions of smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. It also introduces online ComPO, which keeps the offline comparison mechanism while using unlabeled policy generations for reverse-KL control relative to a reference policy.
Following the coverage perspective of preference fine-tuning, the authors provide a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy.
Experiments span Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models. According to the paper, ComPO improves over existing direct alignment methods, including on length-controlled win rates, with pair-level diagnostics offering evidence consistent with mitigating likelihood displacement.