Hugging Face researchers have published a paper detailing LLaDA MoE v2, a 30B-A3B diffusion language model (dLLM) trained from scratch on 23.5 trillion tokens. The work systematically characterizes how optimization hyperparameters, compute allocation, and architecture scale for Mixture-of-Experts (MoE) dLLMs, revealing quantitative differences from scaling trends previously observed in autoregressive (AR) models.
Key findings include that for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model-data allocation, IsoFLOP analysis shows a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains effective and the preferred fraction of activated capacity assigned to shared experts stays stable.
Guided by these insights, the team trained LLaDA MoE v2 using approximately 65% as many pretraining tokens as Qwen3. Despite this, the model approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks.
The paper establishes practical scaling laws and design principles for MoE dLLMs, offering a roadmap for future development of diffusion-based language models. The full paper is available on arXiv.