Existing point tracking models face a fundamental trade-off: they can either follow a sparse set of query points over long horizons, or track all points across only short clips. A new research paper, TrackEverything, proposes a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates.
The approach is grounded in the insight that videos are 2D projections of an underlying 3D world. By working in that 3D space, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead.
The paper introduces three key innovations. First, a voxelization-based de-duplication mechanism at sliding-window boundaries merges co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, tracking is decomposed into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, the authors propose 3D WAFT, which replaces memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud.
According to the paper, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On the TAPVid-3D benchmark, it outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.