Researchers have introduced IntBMoE, a block-conditioned Mixture-of-Experts (MoE) architecture that aims to decouple three quantities that existing MoE designs cannot set independently: participation (how many experts contribute to a token's output), execution (how many experts are actually computed), and materialization (how many expert-sized parameter sets must be built and stored).
According to the paper, sparse routing keeps execution and materialization low but shrinks participation, while dense output-mixing restores full participation at the cost of execution growing with the number of experts. Parameter-merging keeps execution at one expert but its materialization grows with the number of routing decisions.
IntBMoE pairs dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full because every composed expert draws on the entire pool; execution stays sparse because a router sends each token to only a few blocks; and materialization is bounded because the codebook, not the input, fixes how many blocks exist. A component called Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating.
Experiments on image classification show consistent gains over representative sparse and dense MoE baselines, with additional experiments on language modeling and sequential recommendation validating generalization beyond vision. The blocks can be precomputed and cached, making inference computation independent of the expert count when the block configuration is fixed.
The paper reports that IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Code is available at https://github.com/AMAP-ML/DreamX-Rec/.