Researchers have unveiled the architecture behind Qwen3.8-Flash-Next, a sparse mixture-of-experts (MoE) model designed to push the frontier of efficiency and training stability. The model packs 125 billion total parameters but activates only 6 billion per token, a design choice that dramatically cuts computational costs.
According to the paper, Qwen3.8-Flash-Next outperforms its 397B-parameter predecessor (with 17B activated) on eight of fourteen pre-training benchmarks, while trailing on the remaining six by at most 2.6 points. This performance comes at roughly one-third the activated parameters, one-third the training tokens, and about one-ninth the training FLOPs.
The architecture introduces several novel components. Token mixing uses a hybrid of Gated DeltaNet (GDN) and sparse attention, with one full-attention layer in every four. During continued pre-training, these full-attention layers are swapped for Qwen Sparse Attention (QSA), which scores context at micro-block granularity using a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design termed Gated Residual (GR). Additionally, an n-gram embedding layer with tables prefetched from host memory adds capacity outside the backbone.
The authors evaluate each design choice along three axes: loss and downstream benchmarks, computational cost (training, prefill, decode), and impact on hyperparameters and training stability. Notably, they find that loss and downstream accuracy do not always move together—for instance, enlarging the n-gram vocabulary lowers loss monotonically but downstream accuracy saturates.
The combination of the architecture and the Muon optimizer shifts the optimal learning rate and batch size upward, eliminates the need for batch-size warmup, and substantially improves stability under stress tests. The paper argues that loss, benchmarks, efficiency, and stability form a single design problem that, when solved jointly, yields a recipe that is simultaneously more efficient, more capable, and more stable.