Online 3D reconstruction models often struggle with long videos, as regressing poses relative to a fixed first-frame anchor forces extrapolation beyond the training distribution, leading to accumulated drift and geometric collapse. However, researchers at Hugging Face observed that per-frame depth remains stable during this failure, indicating that the backbone's local geometry stays intact while only the global pose head breaks down.
Motivated by this decoupling, they introduce Scal3R, which reformulates online reconstruction as multi-reference relative pose querying. The method uses lightweight learnable tokens, comprising only about 1% of the parameters, injected into a completely frozen backbone via asymmetric attention. This design queries poses relative to multiple past keyframes, and an online pose-graph optimization system with loop closure suppresses long-range drift.
Scal3R achieves convergence in just 8 hours on a single GPU and reduces the average Absolute Trajectory Error (ATE) by over 60% on KITTI compared to the online baseline. It also attains state-of-the-art performance across several benchmarks, including Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes.
This scalable approach enables online 3D reconstruction on kilometer-scale sequences with minimal extra parameters, making it a significant step forward for real-time applications. The project page is available at linjohnss.github.io/scal3r.