Researchers behind the HuatuoGPT line have released a new paper, HuatuoGPT-3: RL-Only Domain Adaptation from Base Models, which explores adapting general-purpose large language models to the medical domain using reinforcement learning alone, without a supervised fine-tuning (SFT) stage.
The authors argue that the dominant SFT+RL pipeline, while offering a convenient cold start, can reduce exploration diversity and adds complexity through multi-stage optimization. Yet pure on-policy RL suffers from a cold-start problem, and mixed-policy RL also falls short: informative tokens in teacher outputs are learned too slowly early on, while stale teacher outputs can hinder later progress. The paper names these two failure modes Gradient Starvation and Teacher-Distribution Anchoring.
To address them, the team proposes One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution, which strengthens learning on informative low-probability teacher tokens, with Teacher Retirement, which discards teacher outputs once the current policy can surpass them.
On medical adaptation, OnePO reportedly achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points respectively. Scaling OnePO produced HuatuoGPT-3, an open-source medical LLM series whose 27B variant reaches 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, which the authors say surpasses frontier models such as GPT-6 Astra.
The paper positions HuatuoGPT-3 as a shift from data-centric medical adaptation toward training-paradigm innovation, following earlier work in the series on medical dialogue alignment, one-stage domain adaptation, multimodal medical understanding, and complex medical reasoning. OnePO was accepted at ICML 2026. Models and code are available at github.com/FreedomIntelligence/HuatuoGPT-3.