A new research paper from Hugging Face investigates what happens when large mixture-of-experts (MoE) models are fine-tuned to reason in a low-resource language—Greek. The headline finding is a null result: accuracy barely changes, and the benchmark itself is too noisy to detect meaningful differences at this scale. Simply changing the random seed swings scores by 7.7 points, more than any data or recipe effect measured.
The real transformations occur where accuracy cannot see. Base models never reason in Greek—0 out of 1,000 traces—even when the question is in Greek. After supervised fine-tuning (SFT), all released checkpoints reason in the question's language on ~98% of items, with one family using 3× fewer tokens. Grammaticality improved on all four models, and general ability stayed within a few points of the base, indicating no catastrophic forgetting.
However, SFT cannot fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit 'think in English' instruction is obeyed less than half the time. Reinforcement learning with verifiable rewards (RLVR), pre-registered before training, fixes the first two outright—format fallback drops from 24% to 2.5%, and leakage from 3.5% to 0.0%—and improves instruction following by 9.1 percentage points, all against a flat random-reward control.
The authors propose six behavioural dimensions to make such changes measurable, each gated to reject metrics that correlate with output length, and they document six failures of their own instruments, each caught by a control. They release five checkpoints, and stress that the instruments, controls, and pre-registration transfer to any low-resource language.