Researchers have introduced Pumpire, a unified benchmark for assessing the metric point-pair distance estimation capability of 3D foundation models operating at both the image and video level, with or without depth priors. The work is presented as a research paper on Hugging Face.
According to the authors, existing evaluation approaches typically assess depth and camera intrinsics separately, or compare point clouds using geometric similarity metrics. Those methods, they argue, cannot directly reflect a model's ability to estimate point-to-point distances. Pumpire instead measures point-to-point distances derived from the reconstructed geometry.
To support this, the team collected pumpire-6k, a large-scale and diverse dataset covering 100 real-world scenes. Each scene contains 64 frames and is annotated with physically measured point-pair distances, yielding a total of 6,400 frames.
Building on this dataset, the authors establish a holistic evaluation protocol that spans image- and video-level 3D foundation models, enabling direct assessment of point-pair distance errors and cross-setting comparisons. They report extensive experiments across 29 baseline configurations of representative 3D foundation models, along with a comprehensive analysis of the results.
The authors frame the benchmark as targeting a more fundamental capability — perceiving and estimating physical scale in reconstructed 3D space — which they say prior evaluation protocols have largely overlooked. A project page is available at pumpire.github.io.