Crimson AI NewsA CrimsonLingua Network service
EN ع
← Back to news
Research paper Hugging Face

QuoteBench: Matched Scores Mask Command-Path Failures in LLM Coding Agents

AI By Crimson AI Hugging Face Papers 21 August 2026 · 00:00 11 views
Share: X Telegram

A new benchmark, QuoteBench, reveals that execution-boundary parsing errors significantly reduce LLM coding agent success, and disclosing the boundary helps recover performance, showing that evaluation must account for deployment configuration.

QuoteBench: Matched Scores Mask Command-Path Failures in LLM Coding Agents

Key points

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. A new benchmark, QuoteBench, shows that matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation.

QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation.

Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models.

GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins.

Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

MetricValue
Tasks56 one-shot
Families14 incident-derived
Success drop (added parser)55.4–73.2 points
Recovery (disclosure)30.4–60.7 points (6/8 configs)
GPT-5.6-sol matched gap-3.6 points
GPT-5.6-sol damage-64.3 points
GPT-5.6-sol compensation+60.7 points
Source
Hugging Face · Hugging Face Papers
Related news
Research paper
Hugging Face 29 Aug 2026

Hugging Face Audit: 110 of 124 AI Evaluations Fail to Support Their Claims

A new commit-bound census of 124 Inspect Evals units reveals that 110 stop before deterministic inference due to missing historica...

4
Research paper
Hugging Face 29 Aug 2026

Aphanta: New Framework Diagnoses When Image Editing Boosts Multimodal Reasoning

Hugging Face researchers introduce Aphanta, a diagnostic framework that evaluates when image-editing intermediates improve multimo...

5
Research paper
Hugging Face 29 Aug 2026

EditaLive! Enables Real-Time Character Video Editing for Live Streaming

Hugging Face researchers introduce EditaLive, a framework for real-time human-centric video editing in live streams, achieving sta...

4