Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as both reconstruction and diffusion latents, bringing strong visual representations into image generation. But they still face a design question: which encoder layers should form the shared latent space used by the generator and the pixel decoder?
That choice involves a trade-off. Shallower layers tend to preserve fine pixel detail, while deeper layers tend to produce better generation metrics. A fixed heuristic layer fusion therefore couples two stages that would benefit from different information. The new FuseReg method replaces heuristic feature selection with training over random subsets of encoder layers.
The authors provide a theoretical analysis of the mechanism: subset sampling explicitly penalizes sensitivity to cross-layer disagreement. The normalized sampling keeps the latent mean unchanged, and under a linear squared-loss surrogate, random subset sampling is shown to regularize cross-layer disagreement, discouraging models from relying too heavily on any particular layer composition.
On ImageNet-256 with DINOv3-L, a single FuseReg decoder reconstructs from full, sparse, and single-layer fusions without retraining, achieving higher PSNR than decoders specialized to fixed fusions. The flexibility also benefits generation: decoder replacement alone reduces unguided gFID from 3.01 to 2.21 with an unchanged RAEv2 DiT-XL generator. Joint regularization of both stages reduces unguided gFID from 13.96 to 9.93 on DiT-Base, and from 2.91 to 2.38 on DiT-XL.
According to the authors, FuseReg distributes reconstruction information more evenly across encoder depth, lowering dependence on individual layers and yielding more stable representations under changing layer compositions. The same trend holds across different pretrained encoder families, and the approach narrows the reconstruction–generation gap without modifying the pretrained encoder.