Long-term memory allows LLM-based agents to retain and reuse information across tasks and sessions, enabling personalization and long-horizon interactions. But persistent memory can also breed sycophancy: agents may over-align with a user's historical beliefs even when those beliefs are inaccurate, outdated, or contradicted by objective evidence.
Existing mitigation approaches assume memory-induced sycophancy stems from biased or incorrect memories, and try to filter such memories at various stages of the memory pipeline. The authors argue this assumption is too narrow. In real-world settings, correct and objective memories can still induce sycophancy, and the same memory may deserve different weight depending on context.
To address this, the researchers propose MemAdapter, a framework that adaptively integrates retrieved memories to support objective and reliable reasoning. It has three components:
- Counterfactual Induction — uses counterfactual reasoning to surface the potential risk posed by retrieved memories.
- Context-Aware Reflection — calibrates how much each retrieved memory should influence inference for the current task via self-reflection.
- Evidence-Based Reasoning — grounds the final response in appropriate evidence while preserving memory's legitimate influence.
Extensive experiments across three benchmarks show MemAdapter consistently improves memory reliability in diverse scenarios, according to the paper. The code is available at github.com/DEEP-JLU/MemAdapter.