Researchers at Hugging Face have introduced InMind, a benchmark designed to measure a critical failure mode in AI agent memory systems: the inability to connect stored facts with indirectly related queries. The work, titled 'Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory,' highlights a fundamental limitation of current retrieval-based memory architectures.
The core issue stems from the assumption that a needed memory will resemble the query that requires it. For example, a user with a tree-nut allergy asking for macaron recipes should trigger a warning about almond flour, yet standard retrieval systems fail because 'tree-nut allergy' and 'macarons' share no surface-level similarity. Similarly, a user with lilies at home seeking cat adoption advice should be warned about lily toxicity to cats, but the system sees no direct link.
To quantify this blind spot, the team created InMind, comprising 125 expert-verified tasks across ten life domains, with 113 grounded in citable public sources. The benchmark includes paired controls to separate three potential explanations: the fact was never stored, the model lacks bridging knowledge, or the fact was stored but not retrieved. The results are stark: when the decisive memory is placed directly in context, the backbone model answers 84.0% of indirect queries correctly. However, when the same memory must be retrieved by six state-of-the-art systems—including HippoRAG 2, A-RAG, A-Mem, Mem0, MemoryOS, and xMemory—accuracy plummets to at most 14.4%, even though these systems recall the same facts on demand with up to 100% accuracy.
Further experiments showed that increasing embedding dimensionality by eight times improved recall for all systems but left the performance gap largely intact. A diagnostic probe that keeps memory visible before the query arrives recovered most of the gap, pinpointing the failure to the query-conditioned retrieval interface itself. The researchers conclude that agent memory cannot be retrieve-only; it must also decide which facts should remain continuously visible, a challenge they call 'routing.'
The findings underscore a critical distinction between notebook-like memory, which stores information passively, and human memory, which continuously shapes judgment. As AI agents become more integrated into daily life, addressing this blind spot is essential for building trustworthy systems that truly 'keep in mind' user information.