Hugging Face has released a technical report introducing DiffusionGemma, an experimental open-weight language model that leverages discrete diffusion to generate text at exceptional speeds. Instead of decoding one token at a time like conventional autoregressive (AR) models, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, effectively bypassing the sequential decoding bottleneck.
The model is built by fine-tuning the mixture-of-experts Gemma 4 model, which has 3.8 billion activated parameters and 25.2 billion total parameters. This compute-efficient approach uses less than 10% of the original AR model's total training token budget, achieved through a two-stage pipeline: first, supervised fine-tuning to teach bidirectional denoising, and second, a combination of reinforcement learning and sampler distillation to jointly improve generation quality and inference efficiency.
According to the report, DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. On average, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU—substantially faster than AR models even with state-of-the-art speculative decoding.
Despite the diffusion fine-tuning, DiffusionGemma retains support for thinking mode, multimodal inputs, and long contexts. It also remains capable of AR generation with only minor performance degradation, suggesting a promising path toward hybrid diffusion-AR decoding in future models.