Score Distillation Sampling (SDS) has become a cornerstone for text-to-3D generation, leveraging pretrained diffusion models to guide optimization of rendered images. However, practitioners have long struggled with structured color artifacts and high-frequency texture noise that degrade output quality. A new paper from Hugging Face, titled PixSDS: Why Latent SDS Makes Noisy Pixels, sheds light on this persistent issue.
The authors identify a specific failure mode: VAE-induced pixel drift. In latent SDS, the optimized image can move along pixel-space directions that the VAE encoder only weakly constrains. As a result, the latent representation appears clean and semantically meaningful, while the actual image accumulates visible artifacts. This disconnect explains why artifacts often appear despite the latent space looking well-behaved.
To validate their diagnosis, the team conducted controlled 2D SDS experiments, VAE-only optimization, and a simplified theoretical analysis. They demonstrate that encoder-like latent objectives can amplify image-space noise when the inverse mapping to pixels is underconstrained. This insight led to the development of PixSDS, a lightweight, VAE-consistent gradient repair method.
PixSDS works by decoding a latent SDS lookahead step and using the decoded image as a clean direction for pixel-space optimization. This reduces motion in VAE-inconsistent directions without retraining the diffusion model, changing the renderer, or replacing the SDS objective. Experiments in both 2D optimization and text-to-3D generation show that PixSDS substantially reduces structured artifacts while preserving semantic content.
The code is publicly available at sevashasla.github.io/pixsds-webpage, and the paper is available on Hugging Face. The authors note that the work originated from a GitHub issue and was developed in their free time, hoping to explain the structured noise artifacts observed during SDS generation.