Researchers at Hugging Face have unveiled Voice Memory, a novel inference-only scheme for agentic speech recognition that requires no weight updates or added parameters. The system introduces a listener-thinker architecture, where a frozen ASR decoder (the listener) and a frozen corrector (the thinker) interact solely through a per-domain memory file (memory.md).
At stream time, the corrector reads the memory and decides per utterance whether to act on the hypothesis or abstain, keeping the 1-best result. Asynchronously, a score-gated optimizer revises the memory through bounded edits, accepting changes only when they strictly improve a held-out score. This design keeps the learned skill auditable and portable, as no model weights are altered.
A key finding is that restraint is the operative skill the loop discovers. Unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news. Voice Memory reduces this harmful-edit rate to 35%.
Across ten HyPoradise domains with an open corrector, Voice Memory lowers weighted word error rate (WER) from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline. Gains are most pronounced in air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%).
The memory transfers across corrector families and adds zero parameters to the inference path. The deployed WSJ memory is just 776 bytes. A demo and example code are available for future studies.