Memory-augmented vision-language model (VLM) agents rely on persistent spatial knowledge to navigate and act in dynamic environments. However, a new empirical study titled "When Memory Lies" reveals a critical flaw: this memory silently becomes stale as the environment changes, and current models often fail to detect the conflict between a confident memory claim and a contradicting observation.
The researchers built a dynamic FrozenLake testbed, pairing a staleness-detection task with a downstream navigation task. They evaluated three closed-source and three open-weight VLMs under both text and image inputs, running 1,800 detection runs and 12,000 text-mode navigation episodes across four LLM navigators at a shared 50-seed scale.
Three key findings emerged. First, text solvability does not imply visual grounding: models that reliably flag stale entries from text still show a wide range of vision F1 scores, from 0.887 down to 0.067 on identical grids. The weakest model continues to make fluent, confident decisions that ignore the image entirely.
Second, consuming stale memory without auditing is a safety liability. In the primary GPT-4o setting, an agent that trusts raw memory dies more than twice as often as an agent given no memory at all. Third, auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, but even oracle stale labels bring no further significant gain on the current grid size. When visual auditing is unreliable, filtering yields no consistent benefit.
The study frames spatial-memory staleness as a safety failure mode and identifies reliable visual grounding and action selection under memory-observation conflict as the central open challenges for memory-augmented agents.