Researchers at Hugging Face have published a comprehensive empirical study on training pixel-space text-to-image diffusion models, addressing a gap in the field where most prior work focused on small-scale or class-conditional settings. The study, titled "An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models," provides a practical recipe for training these models to rival or exceed their latent-space counterparts.
The team first observed that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This motivated a latent-to-pixel strategy: acquire generative priors efficiently in latent space, then transition to pixel space during post-training. The researchers systematically investigated key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule.
Their findings identify a practical recipe that makes pixel-space models match or outperform latent-space models while delivering 3.18 to 4.75 times end-to-end inference speedups. This significant performance gain could make pixel-space diffusion models more attractive for real-world applications where inference latency is critical.
The paper offers useful empirical insights and practical guidelines for future research on pixel-space generation, potentially influencing the direction of text-to-image model development.