Spoken conversational systems are expected to recall details from earlier interactions, but a new benchmark argues that existing evaluations miss most of what makes speech memorable. VoxMem, a benchmark introduced by researchers and published on Hugging Face, tests whether large audio language models (LALMs) can retain acoustic evidence that never appears in a transcript.
The benchmark is built on a two-axis taxonomy: four types of acoustic evidence (speech semantics, speaker identity, paralinguistic cues, and environmental sound) crossed with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal). It contains 3,196 evaluation instances drawn from 34,743 spoken sessions totaling 177 hours of audio, with context budgets ranging from 8K to 64K tokens.
Evaluating 15 LALMs, the researchers found that no model exceeds 40% accuracy at 32K tokens. Proprietary models reach 55.6% on what was said, but only 32.7% on who said it, 20.0% on how it was said, and 21.9% on what was audible. The gap is not a transcription problem: when audio is replaced with exact transcripts, speaker accuracy collapses from 69.8% to 10.3%, while speech-semantics accuracy barely changes (75.9% to 71.0%).
Memory operations sharpen the failure. Models track how a stated fact changes across sessions reasonably well (44.5%), but tracking changes in a speaker's vocal state drops to 3.4%, and tracking a changing background sound falls to 1.2%. Because every question and its evidence stay fixed from 8K to 64K, the decline with longer context comes from the growing history rather than harder questions.
The failures also differ in kind. Speaker errors are mostly binding failures — the right fact attached to the wrong person — while paralinguistic errors are mostly localization failures, where the cue is never retrieved at all. The authors conclude that longer context windows alone do not deliver spoken memory, and that a "transcribe first, remember later" approach is blind to most of what makes a conversation spoken.