Hugging Face has released Loopie, a new family of looped Transformer models that leverage Mixture-of-Experts (MoE) to overcome a longstanding limitation: increasing parameter count typically outperforms looping. The series includes a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters.
Extensive ablation studies show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. The authors compared Loopie against a vanilla 30B-A3B model, demonstrating the effectiveness of their approach.
A novel post-training pipeline equips Loopie with strong reasoning abilities, enabling it to achieve gold-medal performance at the 2025 International Mathematical Olympiad (IMO) and International Physics Olympiad (IPhO) without using external tools.
The paper also references related works such as LoopMoE and Looped State-Space Language Models, indicating active research in this area. The release status of the models and training sets has not been announced.