Reinforcement learning (RL) with verifiable rewards often struggles to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Traditional trajectory-level advantage estimates fail to provide fine-grained supervision. Recent work has explored privileged self-distillation for denser credit assignment, but how to represent sequential credit locally remains unclear.
To address this, researchers from Hugging Face propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through marginal belief revision.
The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. Evaluated on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at 3B and 7B scales, AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.