Researchers behind the GameHorizon Suite have introduced a unified data and evaluation framework designed to measure how well AI models handle gameplay tasks across different temporal horizons. The work, published as a research paper, argues that modern video games offer a measurable testbed for capabilities including visual understanding, instruction decomposition, goal planning, and precise action control.
The suite has three components. GameHorizon-Annotator is an automated pipeline for generating multi-horizon instructions. Using it, the team built GameHorizon-Data, described as the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It contains 5,000 hours of recordings from 21 games, collected by 100 human expert players.
The third piece, GameHorizon-Bench, provides reproducible offline and stepwise online testing. The offline track uses thousands of standardized questions organized into three primary tasks plus diagnostic variants, while the online track checks whether offline scores reflect actual gameplay and pinpoints failures at specific steps in long-horizon play.
Using the suite, the authors evaluated 47 models through more than one million model invocations. They report a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. The suite is intended to serve as a standardized yardstick across model families including VLMs, UMMs, GUI agents, coding agents, and game agents.
The team says the dataset, annotator, and benchmark will be released to support future research. Project links point to a GitHub repository and a dedicated project page.