Long-term memory in multi-party conversations is harder than simple retrieval: systems must track who said what, whom each statement concerns, how participants perceive one another, and how shared and individual states evolve over time. According to a new paper posted on Hugging Face, existing general-purpose LLM memory systems often lose these person and group relations or fail to integrate clues scattered across members, groups, and time.
The authors identify two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, they propose SpeakerMem-R1, a speaker-centered dual-track memory framework. One track stores speaker-labeled verbatim messages; the other organizes derived states into person-level and group-level views. At query time, evidence from both tracks is combined by entity, event, and time.
To reduce attribution and update errors during structured memory construction while enabling local deployment, the team trains a Writer-R1 model using SpeakerLevenshtein and speaker-conditioned GRPO. In a controlled evaluation of 305 questions, reinforcement learning raises the SFT Writer's mean accuracy from 57.38% to 68.20%.
On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, it reaches 62.33%, described as the best reported result among the latest state-of-the-art frameworks, ahead of EverOS (60.08%) and RippleMem (54.75%). The method also scores 70.85% on all 1,986 LoCoMo questions, used as a two-person long-term conversation boundary test.
The paper reports both binary accuracy and token-F1, and ablations indicate that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface. Code and a project page are linked from the paper.