Hugging Face researchers have introduced the Spatial Memory Agent (SMA), a runtime framework designed to enhance the spatial reasoning capabilities of frozen vision-language models (VLMs) without modifying their parameters or relying on external tools during inference. The work addresses a gap in existing approaches, which typically use post-training methods like fine-tuning or call external spatial tools such as depth estimation and 3D reconstruction.
SMA operates by querying a frozen VLM in a verifiable spatial environment, obtaining predictions and rewards, and using verifier-guided reflection to distill compact, transferable lessons from the model's experience. Each lesson is assigned a Transfer Reliability Score (TRS), initially uniform and later calibrated based on retrieval outcomes, providing evidence of future transfer reliability.
During deployment, SMA retrieves relevant lessons using a combination of semantic filtering and similarity-TRS ranking, allowing the retrieved memory to guide the frozen model's inference. This process is entirely parameter-update-free and does not depend on external expert tools at inference time.
In evaluations across five spatial benchmarks and four base VLMs, SMA achieved the highest macro average in every base-model block and the best accuracy in most of the 20 evaluations, demonstrating consistent gains across model scales and environments. The framework establishes a practical path for spatial self-evolution in frozen models, with potential applications in embodied agents, robotic planning, and multimodal assistants.