Automating empirical research has long been a goal of AI, and recent automatic research (AutoResearch) agents have moved it closer to reality as large language models demonstrate the ability to implement solutions independently and learn from execution outcomes. Post-training, especially reinforcement learning, plays a central role in these gains.
In a new paper, researchers identify a fundamental tension when scaling RL for such agents: the two components of every AutoResearch trajectory — agent generation and environment execution — scale very differently. Generation shares compute through batching, while each execution occupies its own exclusive sandbox and real machine time. As trajectories grow, environment execution dominates training cost and becomes the bottleneck.
To address this, the authors propose World Model RL (WMRL), which replaces environment execution with a learned world model. Because a world model can be imperfect, with rewards corrupted by bias and noise, WMRL adds two mitigations: Online Debiasing and Inverse-Variance Denoising, which offset bias and suppress noise respectively.
The paper reports a theoretical proof that both mitigations strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x across various tasks and agent scales while exceeding standard RL baselines. The post-trained 4B and 9B agents reportedly outperform much larger open-weight agents of 48B and 120B on held-out benchmarks.
Beyond AutoResearch, the authors say WMRL transfers to post-training embodied VLA policies, demonstrating the method's generalizability. Code and a project website are linked from the paper page.