Hugging Face researchers have introduced FM-Bench, a novel benchmark designed to evaluate long-horizon decision-making in large language model (LLM) agents. Unlike typical benchmarks that focus on bounded tasks, FM-Bench simulates a 20-year football club management scenario, where agents must navigate cumulative consequences and an environment that reacts to their choices.
In FM-Bench, an LLM agent runs a football club for 20 in-game years, using 26 tools and making roughly 340 to 400 decision stops. The agent drafts a squad on the same budget as rivals, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it. A deterministic engine accumulates each year into a final score, with no LLM judge or human rater involved.
The benchmark includes two tracks: a solo track where 15 frontier models play against a frozen scripted world, and an Arena where the same models plus a scripted anchor compete in a shared 20-year world—the first head-to-head evaluation at this scale. Results show that all 15 models complete every horizon across three seeds, while blind scripted baselines die out in most of theirs. Notably, claude-fable-5 tops the solo board on mean score and the Arena, but the title rotates among ten models.
Key findings indicate that neither scale, price, nor vendor predicts performance. The order settles only late in the horizon, and even the best first-play human lands at the bottom of the model board. What separates models is managerial behavior: higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline. Token spend predicts nothing.
The study also reveals limitations: no model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes—an archive that only grows or a plan rewritten every season. The code is available on GitHub.