Hugging Face researchers have introduced a compute-efficient, two-step hyperparameter transfer framework designed to optimize the training of large-scale Mixture-of-Experts (MoE) models. The approach addresses the prohibitive cost of conventional hyperparameter sweeps, particularly for models trained on trillions of tokens.
The first step adapts Maximal Update Parameterization (μP) for MoE architectures that use Multi-head Latent Attention (MLA) and the Muon optimizer. This adaptation demonstrates that optimal learning rates transfer consistently across models of varying widths, allowing small proxy models to inform larger configurations.
The second step extends this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to optimal values derived from small proxy models on limited budgets, the researchers successfully extrapolated ideal learning rates to massive training horizons, such as 10 trillion tokens, with high fidelity (R²=0.95).
To validate their methodology, the team pretrained a foundation model with 155B total parameters (17B active) from scratch for 10 trillion tokens. The stable training and evaluation results confirm that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs.