Hugging Face researchers have released a new paper introducing UniMoMo, a post-training compression framework designed to accelerate large recommendation models that use sparse mixture-of-experts (MoE) layers. The method converts a trained MoE checkpoint into a smaller standard MoE model under an explicit expert budget, without adding any compression-specific online module.
Traditional approaches often merge experts based on parameter distance, but UniMoMo takes a different route. It groups experts by their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. This is complemented by a layer-adaptive protection mechanism that restricts the merging of high-traffic experts, based on their routing exposure, to prevent performance degradation.
The framework also includes a calibration-aware reconstruction step: after merging SwiGLU experts, a least-squares correction is applied to better recover the original intermediate activations. The final compressed model remains a standard top-(k) MoE, ensuring no extra serving complexity.
Experiments on Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks show that compressing from 8 to 4 experts retains nearly all recommendation quality (NDCG@10 ratios of 99.92%–102.30%) while achieving 1.28×–1.63× speedups on A100 GPUs. A more aggressive 8-to-2 expert, top-1 operating point yields ratios of 98.36%–104.24% and speedups of 1.47×–2.21×.
The authors highlight that UniMoMo is function-aware, routing-aware, and deployment-friendly, making it a practical solution for large-scale recommendation systems and MoE efficiency.