Hugging Face has introduced Mobius-v0, a novel foundation model architecture that decouples knowledge storage from reasoning processes. The design aims to improve knowledge compression and inference efficiency, addressing key bottlenecks in large language models.
The architecture features a globally shared Memory (implemented as a Feed-Forward Network) that stores knowledge vectors, alongside multiple Reasoners (Self-Attention modules) that iteratively perform compositional reasoning. Hidden states serve as both cache and carrier, allowing reasoners to query memory for required knowledge vectors and transmit the retrieved knowledge back to the reasoning operators.
This separation of knowledge and reasoning yields significant efficiency gains. A 7B model trained from scratch with Mobius-v0 achieves downstream scores comparable to a 7B Transformer baseline while using only 62.6% of the baseline's training data. Furthermore, Intern-S2-Mobius, continually pre-trained from Qwen3.5-35B, delivers similar downstream performance with nearly 4x end-to-end inference speedup.
The approach highlights the potential of architectural innovations to reduce training costs and latency, making advanced AI more accessible and deployable in resource-constrained environments.