Discrete diffusion models for categorical generation rely on a corruption kernel that defines the intermediate state space and the reverse prediction problem. A new paper from Hugging Face researchers studies uniform discrete diffusion and asks whether its training objective and reverse transitions can be enriched without changing the underlying categorical corruption process.
The proposed method, named Simplax, introduces an exact Dirichlet-categorical augmentation. This couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the original uniform diffusion process as its categorical marginal. This design yields a tractable Rao-Blackwellized reverse-bridge objective and a corresponding stochastic reverse sampler, while keeping the corrupted categorical state as the denoiser input.
Empirically, Simplax improves the generative perplexity-entropy tradeoff on unconditional OpenWebText generation. On Sudoku, a model trained exclusively on 30-clue puzzles achieves the highest accuracy among compared methods across all evaluated clue densities, including the minimum uniquely solvable 17-clue regime, and also achieves the highest validity in unconditional generation.
The paper is available on Hugging Face and has been recommended by the Semantic Scholar API, with related works including Mean-to-Score Discrete Diffusion, Latent-Kernel Discrete Flow Maps, and UNIFUSION.