Diffusion language models (DLMs) can generate text quickly by predicting multiple tokens in parallel, but they have struggled to match the quality of similarly sized autoregressive (AR) models. A new research paper, ALoDLM: Adaptively Looped Diffusion Language Models, attributes this persistent quality gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict while others need substantially more computation, yet existing DLMs apply uniform computational depth to all unknown positions at every denoising step.
To address this, the authors introduce ALoDLM, which replaces uniform computation with token-adaptive latent recurrence. At each denoising step, the model iteratively refines latent representations and allocates computation according to token difficulty. Tokens that are ready to commit are fed back as discrete context, while unresolved tokens retain and further refine their latent states through additional recurrent passes.
To learn token prediction and computation allocation jointly, the researchers formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). The model was trained at 1.7B and 8B parameter scales.
According to the paper, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average benchmark score at both scales across eleven benchmarks. It also retains fast parallel decoding, yielding what the authors describe as a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.