Researchers at Hugging Face have introduced the full-bandwidth transformer, a new approach that widens the vertical feedback channel in autoregressive transformers. In standard transformers, only the sampled token returns to the bottom of the stack, discarding the rich top-layer hidden state. The new method fuses the previous top-layer hidden state with the sampled token embedding through a gated linear unit, feeding it back as the next input.
This latent feedback allows non-verbalized computation to re-enter the stack with a renewed depth budget, while preserving the standard architecture, KV cache, and language-modeling objective. To train these models without losing parallel teacher forcing, the authors use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability.
Experiments with 1B-parameter models trained on up to 400B tokens show improvements in validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly 1.5 times more tokens, and produce shorter reasoning traces at equal or better accuracy.
The authors acknowledge that scaling-law comparisons are not yet conclusive due to compute constraints, but they highlight the data-efficiency perspective: with limited high-quality data, spending more FLOPs per token can yield better performance than simply increasing token count. They suggest that this paradigm—putting more compute during training via latent feedback—could be a new pretraining direction.