Researchers have introduced MetroLLM-Bench, a 955-case benchmark designed to test language models as the policy layer of a transit kiosk. The benchmark spans six real metro systems — Atlanta, Doha, San Francisco, Taipei, Chicago, and Beijing — ranging from 37 to 414 stations, and covers eleven categories including routing, fare calculation, disruptions, accessibility, and adversarial input.
In each case, a model must call structured tools and return a machine-renderable terminal state containing an outcome, a per-ticket fare quote where applicable, and a kiosk action. Scoring is split into two tiers: Tier 1 comprises fourteen deterministic components, while Tier 2 adds eight semantic-quality components, six of which rely on a language-model judge. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation.
The study evaluates twenty-six models from six vendors, ranking twenty-three of them. On the held-out partition, a 4B Qwen 3.5 student trained via parameter-efficient fine-tuning (PEFT) scored 91.3 on Tier 1, edging out both GPT-5.6 tiers (90.6 and 90.0) and matching GPT-5.4 at maximum reasoning effort (91.4) — all with a 2.6 GB Q4_K_M footprint.
Scaling up did not help at this training scale: 9B and 27B students showed no further Tier 1 improvement over the 4B student. Across four Qwen sizes, the PEFT gain over the corresponding base model fell from +7.03 points at 2B to -0.91 at 27B, with every training seed showing the same direction at every size.
A deterministic rule-based baseline reached 84.6 on Tier 1, leaving the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B topped the composite ranking, and serving configuration alone shifted the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released on GitHub.