Conversational AI systems, especially duplex speech language models (SLMs), have long lacked a memory system that is both accurate and empathetic. To address this, researchers from Hugging Face and Tsinghua University have introduced VoiceMem, a novel memory architecture designed for real-time speech interaction.
VoiceMem employs a dual-brain structure: a parallel informational left brain for factual retrieval and an emotional right brain for affective and persona modeling. This is complemented by streaming memory I/O mechanisms that enable continuous, low-latency memory updates during conversations.
The team also built a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment with interchangeable memory backends. This allows VoiceMem to be integrated into existing systems with minimal disruption.
In experiments and real-world deployment, VoiceMem demonstrated three key advantages: accuracy, emotional personalization, and real-time efficiency. The left brain achieved nearly 30 points higher top-5 retrieval accuracy compared to classical systems like Mem0 at top-200. The right brain, with short- and long-horizon affective attribution and dual-node persona modeling, achieved state-of-the-art performance on three persona benchmarks, improving the aggregate score by 4.29 points over the previous best system.
VoiceMem completes retrieval in just 134 ms, well within standard voice activity detection (VAD) latency, adding no extra conversational delay while maintaining high accuracy and low cost. The project is open-sourced, with code, models, and datasets available on GitHub and Hugging Face.