Hugging Face has released a technical report for Motif 3, a large-scale decoder-only Mixture-of-Experts (MoE) language model. With 314 billion total parameters and 13.2 billion activated per token, the model leverages fine-grained sparsity: each sparse MoE layer contains 384 routed experts, of which eight are selected per token, balancing expert capacity with computational efficiency.
The architecture introduces Grouped Differential Latent Attention (GDLA), which merges grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. Additional innovations include modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction, aimed at improving optimization stability, expert specialization, and inference speed.
Pretraining used approximately 12.5 trillion tokens from diverse sources: web documents, STEM, code, mathematics, multilingual content, and domain-specific corpora. To support stable training at scale, the team employed expert-balancing and numerical-stabilization techniques, along with selective MXFP8 computation, memory-efficient fused kernels, and window-aware context parallelism, enabling context lengths up to 256K tokens.
The post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. This consolidates capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following.
In evaluations, Motif 3 shows competitive performance against leading open-weight models, with strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive benchmarks.