Recent advances in mathematical reasoning for large language models have been driven by post-training methods like Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD). However, these methods typically rely on ground-truth labels, which prevents their use in test-time training (TTT) scenarios where labels are unavailable.
A natural alternative is to replace ground truth with majority-vote pseudo-labels, but this approach is fragile: an incorrect vote can corrupt the teacher and mislead every token. The researchers observed that this failure mode is asymmetric—rollouts that disagree with the pseudo-label are usually wrong, regardless of whether the vote itself is correct.
Building on this insight, they propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors.
Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks. It raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.