As AI models become more capable, the environments used to train them often become too easy, limiting further learning. A new paper from Hugging Face researchers introduces 'environment evolution,' a method that systematically increases task difficulty off-policy to sustain continuous learning signals for terminal agents.
The approach addresses a key limitation of existing co-evolution methods, which rely on on-policy rollouts and thus struggle to provide learning signals as the model strengthens. Environment evolution instead schedules evolved environments generation by generation during training, ensuring that agents are always challenged at the edge of their abilities.
The researchers derived three evolution directions from the multi-turn learning objective and implemented them through a multi-agent harness. Quantitative experiments with several frontier models, including Hy4 preview, Claude Opus 5, and GPT-5.6 Sol, showed that environment evolution consistently produces more difficult environments.
Validation on Qwen3.6-27B and Qwen3.6-35B-A3B using simple long-horizon RL training demonstrated significant improvements: 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively. This suggests that environment evolution can effectively enhance the training of terminal agents without complex on-policy methods.