Reinforcement learning (RL) is a powerful tool for aligning language models, but applying it to small language models (SLMs) in the 70–500M parameter range has often proven unstable. A new paper from Hugging Face, presented at the 2026 IEEE International Conference on Systems, Man, and Cybernetics (SMC), systematically investigates these failures and offers a robust solution.
The researchers trained fifteen (model, corpus) configurations using Proximal Policy Optimization (PPO), spanning models like Pythia-70M/160M/410M and SmolLM2-135M/360M on datasets such as TinyStories, CNN/DailyMail, and Wikitext-103. They identified three reproducible failure modes: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error.
To counter these issues, the team introduced a merge-and-reinitialize adapter technique, switched to float32 precision during PPO updates, and implemented a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. This pipeline led to stable convergence across all experiments.
The paper also proposes a "capacity-headroom hypothesis": PPO performance at the SLM scale depends on both a fluent supervised model (perplexity
All checkpoints, preference datasets, and training scripts have been publicly released to support reproducibility and further research.