Hugging Face has published a research paper introducing DroneCATS, a benchmark designed to evaluate multimodal large language models (MLLMs) as generalist vision-language-action (VLA) agents for drone control. The study, titled "Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control," tests models across four core capabilities: approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet.
The DroneCATS-Agent architecture treats the MLLM as a swappable component, dropping models directly into the drone's control loop with the entire action space declared in the prompt. No fine-tuning or function-calling schemas are used, allowing any MLLM to plug in unchanged. The benchmark evaluates nine models in closed-loop simulation, including GPT-5, Claude Opus 5, Gemini 3.7 Flash, Gemini Robotics-ER 2, the Qwen3.5 family at four sizes, and Cosmos3-Edge-2B.
The main finding reveals a stark paradox: small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. For example, Qwen3.5-9B enters the success radius in 90% of approaching episodes but converts only 35% to successful completions. In fleet commanding, small models blindly copy a single coordinate across distinct views.
The authors conclude that what separates a deployable edge model from a frontier model is not navigation but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs, and DroneCATS is built to measure that distance. Code will be released at https://github.com/naver-ai/DroneCATS.