Looped Transformers reuse a shared block across recurrent loops, producing an intermediate representation at each pass that can be decoded for the next token. Standard decoding, however, discards these earlier states and relies only on the final loop's output.
Researchers propose LoopCD, a training-free contrastive decoding framework that turns earlier recurrent states into built-in weak references. Because earlier loops embody less computation, the recurrence naturally supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. LoopCD contrasts the final prediction with an earlier recurrent pass to guide token selection.
The method comes in two variants: LoopCD-Logits, which operates in logit space with one extra output pass, and LoopCD-Hidden, which works in hidden-state space with zero output overhead. Across four looped Transformer families, both variants delivered consistent gains at full recurrent depth.
Notably, LoopCD-Logits raised Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifted Huginn's HumanEval pass@1 from 22.56% to 31.71%. These gains allow halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%.
By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.