Representation Autoencoders (RAEs) allow diffusion models to operate directly in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not designed for faithful reconstruction and tend to discard fine-grained visual detail, according to a new paper posted on Hugging Face.
The paper notes that finetuning these encoders for image reconstruction does recover such details. But it reports a counterintuitive side effect: the finetuning reduces the effective dimensionality of the resulting representation, and this altered geometry has downstream consequences for generation.
Specifically, the authors show that using the standard velocity prediction objective in flow matching within this high-dimensional space forces the model to fit orthogonal noise directions that lie outside the low-dimensional signal manifold, making optimization inefficient.
This motivates switching to the clean data parameterization, known as x0-prediction, which focuses learning on the underlying signal manifold instead. Across experiments with multiple strong-reconstruction encoders, the authors report that x0-prediction consistently improves text-to-image generation performance.