Open-vocabulary Earth observation (EO) aims to identify geospatial concepts described in natural language rather than relying on a fixed set of labels. However, existing benchmarks often suffer from narrow category vocabularies and limited query forms, hindering progress in this field.
To address this gap, researchers have introduced OVEarth-Bench, a new benchmark that extends evaluation along two key dimensions: category breadth through broad hierarchical category coverage with positive and negative expressions, and query diversity through vocabulary, referring, and reasoning queries. The benchmark supports both mask and box localization under a unified zero-shot protocol.
The evaluation of a wide range of general and EO-specific methods reveals several important findings. First, the performance of current methods remains limited, though broader category coverage leads to more stable model rankings. Second, multimodal large language model (MLLM)-based methods achieve the strongest overall performance. Third, EO-specific methods generally underperform general models and rarely match the strongest methods.
These findings offer guidance for future open-vocabulary EO method design and underscore the need for more realistic, diverse, high-quality, and large-scale benchmarks. The data and evaluation package are publicly available at https://earth-insights.github.io/OVEarth-bench.