Researchers have unveiled MMDiff, a new framework designed to make the inner workings of multimodal large language models (MLLMs) more transparent and controllable. The framework leverages multimodal sparse autoencoders (SAEs) to isolate, detect, and control specific features that drive visual understanding and safety behaviors in these models.
While sparse autoencoders have been used to decompose hidden states into interpretable feature directions, they have not readily isolated which features are changed by multimodal training or been directly useful for targeted control. MMDiff addresses this by training multimodal SAEs and turning them into feature-level interfaces for discovering and controlling multimodal behavior.
The framework supports three key uses: feature isolation by diffing a base-LM SAE against its multimodal-adapted counterpart, task-specific feature detection via per-token contrastive firing analysis, and feature-level control by causally removing or steering discovered feature directions. The researchers applied MMDiff to three MLLM families: LLaVA-MORE, PaliGemma 2, and InternVL3.5.
In evaluations on visual-spatial understanding, multimodal safety, and OCR, MMDiff discovered sparse, causally specific features. Removing these features selectively degraded target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduced attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering the features improved spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline.
The results suggest that multimodal SAEs can serve not only as interpretability tools but also as mechanisms for auditing, steering, and controlling MLLM behavior toward safer and more capable generations. The project page is available at https://pixl.cs.ox.ac.uk/mmdiff/ and the arXiv paper at https://arxiv.org/abs/2608.09928.