A new research paper from Hugging Face, titled "Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing," challenges the conventional geometric interpretation of sparse mixture-of-experts (MoE) language models. The authors argue that the benefits of routing tokens to multiple experts are not simply due to co-selected experts contributing distinct representation directions.
To disentangle the effects of route coherence, candidate quality, and candidate-by-context interaction, the researchers introduce the Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled 2x2 factorial design. They also employ frozen-route interventions and a controlled Top-k study to assess functional value.
The study's findings are organized around three paired contrasts. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, in all 39 factorial cells across OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival, but the actual prefix narrows this advantage in every case, with all interactions negative and every 95% confidence interval below zero.
Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive. A controlled training study also favors Top-2 over Top-1 in all three seeds.
The authors call this joint pattern "coherent overlap": routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. This distinction clarifies why geometric similarity alone cannot determine redundancy or pruning value, with implications for MoE routing, expert redundancy, and pruning.