Masked diffusion models (MDMs) generate text by unmasking several tokens at each denoising step, but their reverse process is typically factorized across positions. According to the authors, this factorization limits sample quality precisely in the few-step regime, where diffusion's speed advantage over autoregressive decoding matters most.
Prior work attempted to capture cross-position correlations by introducing a continuous Gaussian latent trained as a variational autoencoder. The paper notes that such approaches are prone to posterior collapse, in which the latent is silently ignored by the model.
The proposed method, Enhanced Mixture-of-Experts (E-MoE), takes a different route. It builds the reverse process as a mixture of factorized distributions over a discrete shared latent, defined by the expert-routing decisions of a Mixture-of-Experts backbone. The authors state that this does not increase active parameters relative to the factorized baseline.
The paper reports that E-MoE improves few-step generation over factorized baselines across synthetic multi-modal benchmarks, binarized MNIST, and LM1B. On the synthetic benchmarks and binarized MNIST, the authors say the method recovers multimodal structure that factorized baselines fail to capture.
The work was posted on Hugging Face Papers, where a Librarian Bot listing surfaced related recent work on diffusion language models, including LLaDA MoE v2, distribution matching distillation for continuous diffusion LMs, and simplex relaxation for discrete diffusion.