Researchers have introduced Urbex, an agentic vision-language framework for grounding objects in city-scale 3D scenes. According to the paper, most existing approaches frame the problem as candidate ranking, which depends on precomputed instance proposals and scales poorly in large outdoor environments.
Urbex takes a different route: it treats grounding as active spatial search. The scene is represented as an interactive multi-view environment in which a vision-language model agent can query landmarks, zoom into local regions, render oblique views, and finally commit a grounded bounding box.
The team optimizes this tool-use policy with reinforcement learning on the CityRefer dataset, using smooth localization rewards alongside lightweight shaping rewards to encourage efficient exploration.
Experiments reported in the paper show Urbex achieving the best Hit@1 and IoU on CityRefer. The authors also note that the method supports candidate-free grounding without ground-truth boxes and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer are said to indicate improved cross-scene robustness.
The project is released publicly, including code, a 4B model, data, and a Colab demo.