Reinforcement learning (RL) has become a key component in enhancing the reasoning capabilities of large language models (LLMs), enabling them to solve complex math problems, write code, and reason over multimodal inputs. However, stable policy optimization remains challenging due to variance in token-level importance ratios, particularly in large Mixture-of-Experts (MoE) models.
Existing methods like GRPO and GSPO use hard clipping to control instability, but this approach discards useful gradient information and struggles to balance stability and sample efficiency. To address these limitations, Alibaba's Qwen team proposes Soft Adaptive Policy Optimization (SAPO), a novel RL method that replaces hard clipping with a smooth, temperature-controlled gating function.
SAPO offers several advantages: continuous trust regions that avoid discontinuities, sequence-level coherence similar to GSPO but without discarding entire sequences, token-level adaptivity to selectively suppress problematic tokens, and an asymmetric temperature design that treats positive and negative advantages differently. This design stabilizes training and improves sample efficiency.
In experiments, SAPO was tested on mathematical reasoning using a cold-start model fine-tuned from Qwen3-30B-A3B-Base. It maintained stable training longer than GSPO and GRPO-R2, achieving higher Pass@1 scores on benchmarks like AIME25, HMMT25, and BeyondAIME. SAPO also does not require routing replay, simplifying RL pipelines.
Large-scale RL experiments with Qwen3-VL models showed consistent performance improvements across various model sizes and architectures, including both MoE and dense models. SAPO outperformed GSPO and GRPO-R2 under the same compute budget.
The team believes SAPO will become a foundational component of RL training pipelines for LLMs, offering a practical way to stabilize and enhance training. The full paper is available on the Qwen research page.