Latent generative models typically follow a two-stage pipeline: first, a variational autoencoder (VAE) is trained for reconstruction, and then a generative model is trained on the frozen latent space. However, latents optimized for reconstruction are not necessarily generation-friendly, prompting interest in joint end-to-end training. Yet, direct end-to-end training has been challenging due to latent collapse and a generation-reconstruction conflict.
In a new paper, researchers revisit this problem by analyzing how different objectives shape the latent space. They identify two key insights: the entropy term in the Kullback-Leibler divergence objective is crucial for preventing collapse, as reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Additionally, reconstruction and generation exhibit asymmetric learning dynamics—reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize.
Based on these insights, the team proposes GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. This approach achieves the first direct end-to-end training without latent collapse.
The method is validated with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With the end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. The framework also extends to shared visual latents for generation and representation learning, and to continuous unified text-image generation, demonstrating its generality across generative priors and modalities.