Researchers from SLAI have introduced SLAI T-Rex, a full-stack optimization framework for post-training trillion-parameter Mixture-of-Experts (MoE) models on Ascend NPU SuperPOD clusters. The work addresses critical system-level challenges such as memory pressure, communication overhead, and inefficient kernel execution that arise when training large-scale models on non-GPU hardware.
The framework employs a hierarchical approach spanning model-level parallelism, computation-communication orchestration, and low-level kernel optimization. On the DeepSeek-V4 model family, SLAI T-Rex achieves 34.22% Model FLOPs Utilization (MFU), a 2.93x improvement over the open-source baseline recipe, while maintaining training stability.
Building on this infrastructure, the team developed a domain-specific workflow for Operations Research (OR) tasks. They created curated Continuous Pretraining (CPT) and Supervised Fine-Tuning (SFT) data pipelines, combining collected domain resources with solver-verified synthetic optimization documents. The resulting dataset includes 10,000 high-quality SFT samples across four task categories and three problem representations.
The specialized model, DeepSeek-V4-Flash-OR, achieves the highest average zero-shot Pass@1 score among evaluated models at 71.81%, outperforming GPT-5.4-Mini by 3.98 percentage points and the base DeepSeek-V4-Flash model by 11.27 percentage points. The model and code are publicly available on ModelScope and GitHub.