Reinforcement learning with verifiable rewards (RLVR) has become a key technique for improving the reasoning abilities of large language models. A newly posted paper on Hugging Face, titled Bellman Policy Optimization, proposes a critic-free approach aimed at that setting.
The method, abbreviated BPO, is derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, the authors use the Bellman equations to reformulate PMD as a trajectory-level objective, which avoids the need to estimate state values at intermediate states. The paper states that the reformulated objective shares the same unique optimal solution as the original PMD objective.
From this reformulation, the authors derive a practical BPO loss by approximating the trajectory-level objective. According to the paper, the loss replaces the importance-sampling ratio used in Group-Relative Policy Optimization (GRPO) with a smoothed ratio of complementary token probabilities, described as a mismatch-correction weight.
The paper reports experiments on mathematical reasoning benchmarks demonstrating the effectiveness of BPO. It also notes that BPO and a separate work, Score Centering Stabilizes Off-policy Reinforcement Learning (SC), share an identical core gradient obtained through different derivations: BPO derives it via Policy Mirror Descent and the Bellman equations, while SC derives it by removing gradient drift. The two approaches then diverge, with BPO using a binary approximation to the KL divergence between the sampling and policy distributions plus additive smoothing, and SC using a top-k approximation.