Large language model agents are increasingly deployed on long-horizon tasks, where a single run may involve dozens of intermediate decisions — which hypothesis to test, which implementation to build on, which direction to pursue. A new paper posted to Hugging Face argues that the ability to make these choices well deserves its own name: the taste of an agent.
Existing benchmarks measure whether an agent ultimately succeeds, but none isolates the quality of the decisions made along the way. To fill that gap, the authors built Taste-Bench, a benchmark of 502 decision forks mined automatically from agent trajectories on SWE-bench Pro and METR AI R&D tasks. Each question presents a fork — a point where multiple directions are available and one leads to a better outcome — and the model must choose without seeing what happens next. Labels are derived from what actually occurred later, with 98.8% agreement with human review.
Evaluating 14 frontier models, the authors found that the best model answers only 59.7% of the questions correctly, against a random baseline of 25%. Accuracy falls sharply from 62.3% to 21.0% as the deciding evidence moves further into the future, and increasing the reasoning budget does not improve results.
The paper also shows that taste can be trained. By distilling the judgment of a teacher model that has seen the outcome into a student, the authors lifted Qwen3.6-27B from 30.0% to 47.9% on unseen tasks. The student's advice also raised end-to-end success on held-out SWE-bench Pro tasks from 14.6% to 33.7%.
The dataset and code are publicly released, and the authors note that the generator model is instructed to verify that a target decision genuinely contributes to final success, so lucky trajectories are dropped rather than rewarded.