Researchers have introduced VoxPolyMem, an interaction-aware multimodal memory framework designed for long-term memory in multi-party spoken conversations. According to the paper, existing long-term memory research has largely focused on dyadic text or image-text exchanges, leaving multi-party spoken settings underexplored.
The framework combines incremental speaker identification with a memory hierarchy made up of interaction memory, fact memory, and participant profiles. This structure is intended to preserve conversational content, identify participants across sessions, and retain who speaks to whom.
Retrieval is formulated as sequential decision-making, where an agent rewrites queries and selects retrieval tools and memory layers based on accumulated evidence to fill information gaps. The authors also propose Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition.
To evaluate the approach, the team constructed VoxPolyBench, covering memory evolution, personalized answering, memory retrieval and reasoning, and interaction reasoning and attribution in multi-party spoken conversations.
VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points. On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively, exceeding the strongest evaluated public memory baselines by more than 8 points each. Code and datasets are available at the project's demo page.