Researchers have introduced Latent-to-4D, a novel approach for generating dynamic 3D scenes (4D) directly from video diffusion latents. The method aligns a video latent with the token grid of a pretrained 4D decoder, refining it through frame-wise and global spatiotemporal attention. This bypasses the traditional RGB reconstruction step, avoiding distribution mismatch and error propagation.
Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a specific video generator to predict geometry directly. The former suffers from error accumulation, while the latter ties 4D prediction to a particular generator, often requiring retraining when the generator changes. Latent-to-4D instead leverages the shared variational autoencoder (VAE) of video models to create a reusable interface.
Trained on roughly 1,000 existing reconstruction clips, a single checkpoint of Latent-to-4D transfers unchanged across multiple video diffusion transformers within the same VAE family. This demonstrates significant flexibility and efficiency compared to prior approaches.
On the Text4D-200 and I4D-200 benchmarks, Latent-to-4D outperforms matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88–3.45 and 5.81 points, respectively. Human raters also preferred its outputs for geometry, temporal stability, and overall quality.