Hugging Face researchers have introduced StateAct, a novel multi-agent framework for long-horizon computer-use tasks that prioritizes program state over pixel-level perception. The core insight is that screenshots are a lossy representation of underlying program state—files, backends, and DOM—which code can inspect and modify directly.
StateAct's main agent operates primarily through code, accessing program state directly. A dedicated GUI subagent handles screenshot-and-click interaction only when necessary—just 28 of 108 tasks and 1.1% of main-agent steps. An independent finish gate verifies saved results for structural failures like missing or unsaved output.
On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.6% partial success, at ~9x lower cost ($7.8 vs $72 per task, 100K vs 224K tokens). A code-only variant (no GUI subagent) reaches only 45.9% partial success, below the screenshot-based baseline.
The framework also improves performance across five benchmarks: OSWorld-Verified (80.9→81.9), WindowsAgentArena (41.6→50.6), AndroidWorld (69.0→81.9), and MobileWorld (51.3→70.1). When applied to Hugging Face's internal 31B SFR-CUA model, results leap across all benchmarks—e.g., OSWorld 2.0 from 7.6% to 43.2%.
StateAct demonstrates that grounding action, verification, and memory in program state—termed state-grounding—shifts the bottleneck from perception to reasoning. Failures depend more on what the agent thinks than on what it sees.