A new research paper from Hugging Face investigates the fundamental differences between supervised fine-tuning (SFT) and reinforcement learning (RL) in multi-task reasoning for large language models (LLMs). The study reveals a striking contrast: SFT suffers from severe task conflicts under multi-stage training, while RL allows different task capabilities to coexist and improve stably.
Through parameter-level analysis, the researchers observed that RL induces sparse and approximately orthogonal updates across tasks, whereas SFT updates are denser and more interfering. The paper provides a theoretical explanation by analyzing multi-task gradient interference, showing that interference in SFT is norm-limited—scaling with absolute gradient magnitude—while interference in RL is variance-limited, bounded by gradient variance from advantage normalization and on-policy optimization.
This variance-bound leads to near-orthogonal optimization directions across tasks, enabling stable coexistence. Based on this insight, the authors propose Parallel-RL, a paradigm that decouples multi-task RL into parallel task-specific training, significantly improving efficiency and flexibility.
The findings offer a deeper understanding of why RL excels in multi-task settings and could guide future training strategies for LLMs.