Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. A new paper, TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning, examines why native NVFP4 RL can become unstable even when learner–sampler mismatch is controlled by standard magnitude-based corrections.
The authors find that mismatch magnitude alone is insufficient to characterize risk. What also matters is whether the policy-gradient update contracts or amplifies the existing mismatch. In both Qwen3-4B and Qwen3-30B-A3B, a directional imbalance emerges before the global mismatch distribution becomes severe, with mismatch-amplifying negative-advantage tokens becoming concentrated in a small fraction of response segments.
Motivated by this observation, the paper introduces TRIAGE, which performs segment-level diagnosis, selectively attenuates mismatch-amplifying updates, and applies bounded repair to residual severe mismatch. Importantly, TRIAGE preserves native NVFP4 weight-and-activation 4-bit (W4A4) execution on both the sampler and learner, rather than replacing it with fake quantization.
Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and full-precision-level performance across five mathematical reasoning benchmarks. Native NVFP4 with TRIAGE provides up to 2.30× rollout throughput and 1.30× end-to-end RL speedup over BF16.
The authors hope this work provides a useful perspective for understanding and stabilizing execution mismatch in low-precision LLM RL.