Training GUI agents typically requires large volumes of high-quality interaction trajectories so the agent can learn how software environments respond to actions, maintain state, and support multi-step workflows. According to the paper, the diversity of available trajectories is limited by the applications, interface states, and workflows that can be accessed in the underlying environments, and expanding that coverage means deploying increasingly diverse and complex software — with specialized applications adding installation, configuration, and runtime costs.
To address this, the authors introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments.
The pipeline works in stages: AutoGUIWorld samples initial GUI scenes from structured specifications covering operating-system context, visual appearance, and interface state, then generates tasks conditioned on those scenes. A planner specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering then yield 79,266 spatially annotated step-level training samples spanning Ubuntu, Windows, macOS, and Chrome.
The authors report that fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8%, and raises the task success rate on ScienceBoard from 14.0% to 32.2%. They conclude that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.
The project provides a GitHub repository and a full paper report linked from the Hugging Face paper page.