Looped transformers reuse layers for latent computation, offering parameter efficiency, but whether looping actually improves test-time scaling as outputs grow longer has remained largely unexplored. A new research paper examines this question by post-training looped transformers and measuring the accuracy-compute slope — the accuracy gain per doubling of test-time decoding FLOPs.
The authors find that existing looped transformers often show steeper slopes than their non-looped baselines, yet still underperform those baselines at matched compute. Their analysis indicates that fixed-depth looping spends extra iterations on every token, even though many tokens gain nothing from additional iterations.
To address this, they propose TaH2, which concentrates extra iterations on the tokens that actually benefit from looping. The method jointly post-trains the backbone and an iteration decider using lookahead depth supervision, relying on online labels that indicate whether further iteration improves the prediction.
On challenging AIME benchmarks, TaH2 improves the accuracy-compute slope by 53% (2.74 vs. 1.79) over the non-looped baseline and exceeds that baseline's peak accuracy by roughly 3.4 points at matched test-time compute. As maximum iteration depth grows, existing looped models largely plateau, while TaH2's advantage over the non-looped baseline continues to increase — from +2.8 points at depth 2 to +3.9 points at depth 8.
The authors state that TaH2 improves both the efficiency and the attainable accuracy of test-time scaling, and have released their code at github.com/thu-nics/TaH.