Generating an egocentric video from a single exocentric recording is a hard case of novel view synthesis: the two cameras share little overlap, and much of the target view is never observed. A new paper titled LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation, featured on Hugging Face, tackles this problem by questioning what a video diffusion model should actually receive as its condition.
Existing state-of-the-art pipelines reconstruct the scene explicitly — estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. Because this deterministic mapping assigns each pixel to a single reprojected location, it preserves texture but turns depth errors into misplaced content.
The authors propose a lifting-free alternative: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly, without depth, point clouds, or reprojection. Cross-view correspondence is resolved internally by the model. Its probabilistic mapping averages each region over candidate source locations according to a learned correspondence distribution, preserving structure while averaging away fine texture.
The paper argues this trade-off suits a diffusion generator, whose denoising training is well suited to restoring detail, so an effective condition should prioritize structural alignment over sharpness. The concentration of the correspondence distribution also yields a per-region confidence score, used to mask low-confidence regions and to steer the generator toward high-confidence areas during early layout-forming denoising steps.
According to the authors, the approach consistently outperforms the state-of-the-art explicit pipeline and generalizes to other datasets without retraining. In their framing, the synthesizer supplies view structure while the diffusion model supplies detail.