Layer dropout, a technique once common in transformer training but largely abandoned in large language models (LLMs), is making a comeback. A new paper from Hugging Face shows that with the right configuration, layer dropout can significantly improve both training efficiency and inference speed without harming accuracy.
The study, which involved over 2,400 experiments on models ranging from 271 million to 8.2 billion parameters and datasets up to 160 billion tokens, found that previous reports of accuracy degradation were due to suboptimal settings, not a fundamental flaw. By optimizing four key ingredients, the researchers achieved lower validation loss at the same training FLOPs, or up to 25% FLOP savings for the same loss.
The four critical components are: a scaling factor of 1/(1−p) for hyperparameter transfer across dropout rates, dropping whole transformer blocks per-sequence rather than per-batch or separate attention/FFN layers, an Increasing Layer Distribution (ILD) that protects early layers while dropping later ones more aggressively, and a Decreasing Time Schedule (DTS) that starts with maximum dropout and decays to zero over training.
Beyond training, layer dropout enables 'free' post-training depth elasticity, including early exit and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. The benefits compound with scale, as larger models tolerate higher maximum dropout rates—up to 0.99 for the 8.2B model.
All experiments were conducted on Cerebras CS-3 systems, and the findings suggest that layer dropout should be a standard part of state-of-the-art LLM training recipes.