Pretrained transformers appear to use only a small fraction of their depth when following reference chains in context, according to a new paper titled "Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It." Across thirteen base models ranging from 0.6B to 32B parameters, the researchers found that models reliably follow only 1.4 to 3.6 lines of a chain such as K = apple; B = K; D = B; print(D) when asked to answer directly without chain-of-thought.
The team reports that adding pretrained loops yields little improvement, and even doubling model depth — from OLMo-3 7B to 32B — leaves the reach at roughly 2.6 lines. The fix, they say, is surprisingly small: a task-trained rank-8 LoRA applied at one early layer, with all base model weights frozen. On Qwen3-8B, a LoRA at layer 14 with 65,537 parameters raised exact accuracy on 24-line chains from 15.5% to 99%, while a longer-trained version reached 50 lines in a single forward pass.
The paper describes a relay mechanism: the LoRA acts on each token independently and moves no information between tokens, but it starts a process in which program lines pass their chain identity through frozen middle layers 16–22. Frozen heads then read progressively further up the chain. Cutting each line's attention to its parent in layers 14–22 drops accuracy to chance, while the same cut in layers 23–29 barely matters.
Placement proves critical. The same recipe placed at layer 20 reaches 20.5 lines, but at layer 21 it falls to 5.2 lines. A frozen-model measurement located this limit within a preregistered tolerance in three of four held-out models. In looped models, a LoRA applied every loop in Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight.
On multi-hop question answering, separately trained early-layer LoRAs added 9.4–17.9 EM across three standard models on MuSiQue with gold paragraphs. The authors conclude that default answers understate the computation accessible through a tiny edit, and have released code and an interactive demo.