Researchers have introduced Hierarchical Continuous Diffusion Language Models (HC-DLM), a new approach that seeks to overcome a structural limitation shared by discrete diffusion language models. According to the paper, when such models decode tokens in parallel, each token is sampled independently from its marginal distribution, which breaks the statistical dependencies among tokens decoded together.
Continuous diffusion language models avoid this issue by denoising a shared continuous state, but their denoiser only sees that state, so nothing ties it to a valid token configuration until final decoding. HC-DLM addresses this by coupling discrete token generation with a continuous latent trajectory inside a single denoising process, with a training objective derived from a variational bound on the token likelihood.
Unlike recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and fed back as a scaffold for the next latent update.
The authors evaluate HC-DLM on structured reasoning (Sudoku), mathematical planning (Countdown), and language modeling (LM1B). At matched model size, the method improves over both discrete and continuous diffusion baselines in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B.
A project page is available at https://hc-dlm.github.io/.