Visual generators can produce photorealistic frames without maintaining a consistent 3D scene. A new research paper argues this is not only a modeling problem but also a representation problem: generators typically evolve appearance-centric latents, while perception models recover geometry in a semantically rich space that encodes cross-view structure.
To address this, the authors propose the Geometry-Native Autoencoder (GAE), which reparameterizes the features of a geometry foundation model into a compact latent space for generation. Rather than treating geometry as an additional output, GAE makes geometry the native substrate of the latent state.
The GAE latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks. Using camera trajectories as conditioning, the video model generates a unified state that jointly decodes into RGB, depth, and 3D geometry.
In controlled comparisons that hold the generator and training protocol fixed, replacing the latent with GAE improves both visual quality and independently measured 3D coherence: FVD falls by 12.7% and 23.1% on RealEstate10K and DL3DV respectively, and camera-trajectory error is halved on RealEstate10K.
The authors conclude that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation. Code and a project page are publicly available.