Diffusion language models (DLMs) are emerging as an alternative to autoregressive text generation, and continuous DLMs in particular apply latent diffusion over continuous text embeddings. A new paper shared on Hugging Face asks a practical question: which embedding produces the best latent space — that is, the most diffusible one?
To answer it, the authors searched across different embeddings and found that scaling the embedding model to stronger versions within the same family — from T5 to T5Gemma-1 to T5Gemma-2 — greatly improves generative performance. According to the authors, replacing the T5-small embeddings used in recent ELF models with T5Gemma-2-270M reduced Gen. PPL by roughly 40%.
However, the paper notes that scaling alone is not enough. Raw T5Gemma-2 embeddings are highly discriminative, separating even plausible alternative words, which makes generation vulnerable to imperfect sampling. As a result, continuous diffusion often fails to land on any valid embedding and ends up at an invalid one.
To address this, the researchers distill T5Gemma-2 into a student encoder that learns the teacher's decoded probabilities as soft labels. This pulls alternative embeddings closer together while preserving the encoding-decoding mechanism, producing a more connected and diffusible latent space that improves on the vanilla T5Gemma-2 embeddings.
The authors report that their medium-sized DLM reaches Gen. PPL 17.8 (against real-text PPL 15.4) at real-text entropy on OpenWebText, outperforming GPT-2-M on Gen. PPL. Code and a project page have been released alongside the paper.