Researchers have introduced LatentLoop, a method for scaling looped Mixture-of-Experts (MoE) models by allocating computation unevenly across repeated visits to the same shared block. The work appears as a paper on Hugging Face and targets the trade-off between parameter reuse and sparse computation under fixed budgets.
The starting point is an observation from prior work: different loop visits can serve different roles even when they share parameters. To probe this, the authors learned separate residual coefficients in a model that applies a shared block twice, and found the first visit operated at a smaller scale. Fixed-coefficient experiments reinforced the pattern — shrinking the first visit's scale preserved or improved performance, while applying the same reduction to the second visit hurt it.
That asymmetry led to the proposed design. LatentLoop compresses the first visit's sequence, retains full resolution in the second, and reallocates the computation saved to increase the model's hidden size. The method also learns how strongly the first visit updates the full sequence, and shares MoE weights across visits while using separate attention parameters to handle the differing sequence resolutions.
Across 15 configurations spanning 1.3B to 7B parameters, with the largest budget exceeding 1021 FLOPs, LatentLoop consistently lowered validation negative log-likelihood (NLL). At a fitted NLL of 1.988, the authors estimate compute savings of 19.0% relative to a vanilla MoE and 7.9% relative to the looped baseline. The approach also improved average scores on nine benchmarks at all sizes in both 0-shot and 5-shot settings.
The authors conclude that asymmetric compute allocation is a promising direction for looped MoE models, framing the result as an efficiency gain under matched budgets rather than a purely architectural change.