Research planning is a critical capability for AI scientists, but it lacks verifiable answers, making reinforcement learning (RL) difficult due to the absence of a clear critic. Rubrics from scientific papers can serve as that critic, yet existing pipelines derive both the question and criteria from the same content, allowing models to game the reward through paraphrase. Additionally, rubrics are often compressed into a single scalar per rollout, losing granularity.
To address this, Hugging Face researchers propose PaperGym, a unified framework that transforms each research paper into a complete training environment. PaperGym leverages the paper's structure: the research question is synthesized from the goal and background, while the evaluation criteria are derived from the method and experiments. This separation reduces criterion leakage to just 3.7%, compared to 11.90%–34.10% in existing datasets.
The training process uses the rubric twice: first as privileged context for the OPSD self-teacher, then as the reward signal for GRPO. Across Qwen3 models (1.7B, 4B, 8B), this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points respectively.
With the recipe fixed, models trained on the PaperGym-20k corpus win 58.1% of three-way comparisons, versus 28.2% for RubricHub Science. The trained Qwen3-8B achieves 73.48 on ResearchQA, surpassing the much larger Kimi K2.6. The team releases the pipeline, the 20,000-instance corpus, and two benchmarks: PaperGym-Innov and PaperGym-Design.