Researchers from Hugging Face have introduced SMELT (Sparse MoE Transformer, middle layers Loop Twice), a new architectural recipe that improves the efficiency of sparse Mixture-of-Experts (MoE) Transformers by looping the middle half of the network's layers twice. Unlike previous studies that compared looped models at a fixed size—thereby conflating architectural benefits with extra compute—this work carefully matches per-token FLOPs, total non-embedding parameters, and KV cache against a standard unlooped baseline.
The team scaled SMELT across four model sizes, up to 54 billion non-embedding parameters, and fitted a separate Chinchilla-style scaling law for each architecture. The results show that SMELT's validation loss drops faster with compute, saving between 6.8% and 18.0% of training FLOPs on the compute-optimal frontier. These savings are not merely theoretical: the performance advantage transfers to downstream benchmarks, with the largest gains observed on code-related tasks, and the benefit grows with longer sample sequences and more in-context examples.
Mechanistic analysis reveals a plausible explanation for the improvement: during the second pass through the looped layers, the model reduces the attention sink phenomenon and redirects attention mass toward content-relevant tokens. This inductive bias may underlie the observed performance gains, suggesting that looping can serve as a practical method to turn depth reuse into measurable improvements, even under strict budget matching.
The findings offer a concrete recipe for AI practitioners looking to optimize training efficiency without sacrificing model quality, particularly in MoE architectures that are increasingly common in large-scale deployments.