Public benchmarks are essential for measuring Automatic Speech Recognition (ASR) capabilities, but they also create a risk: models may be optimized to perform well on these benchmarks without generalizing to real-world data. A new paper from Hume AI, titled "Towards Quantifying Benchmark Optimization in ASR Models," presents a methodology to quantify this phenomenon.
The researchers identify three families of behavioral probes that reveal models' abilities to reproduce benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. They find that the highest-scoring open-source models output verbatim reference transcript spans even when the audio is contradictory, masked, or ambiguous.
Using mechanistic probes, the authors show that models respond to narrow acoustic cues to override faithful audio representation in favor of a benchmark-optimized policy. They also demonstrate that this behavior can be causally manipulated via low-rank linear steering or by appending audio to the end of a segment.
The production version of this problem is that benchmark word error rate (WER) stops predicting much once audio is not clean read speech. The gap appears on accented speakers, overlapping talk, and telephony codecs, which are underrepresented in standard sets. A useful tell is that a model's advantage over the field collapses when held-out audio recorded through a different chain is used, even at matched nominal difficulty.
Overall, the results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.