Researchers at Hugging Face have published a paper proposing a structured approach to scaling large reasoning models (LRMs) beyond the limits of human supervision. The work addresses the challenge of improving models in open-ended and agentic tasks, where traditional reinforcement learning with verifiable rewards (RLVR) falls short due to the difficulty of obtaining reliable rewards and the impracticality of direct human oversight at scale.
The paper introduces a five-level ladder, from L0 to L4, that categorizes the degree of human control remaining in the learning process. This ladder spans two key dimensions: the reward axis, which evolves from per-instance human judgments to reusable verifiers and autonomous rewards, and the experience axis, which progresses from human-curated tasks to self-generated curricula and autonomous co-evolution.
The authors also highlight significant risks associated with increasing autonomy, including reward hacking, feedback drift, curriculum collapse, and environment errors. To address these, they propose evaluating three complementary objects: policy capability, feedback fidelity, and experience quality.
This analysis offers a structured account of current methods for scaling LRMs beyond human supervision and outlines open problems in developing self-sustaining learning systems toward superintelligence. The team maintains a continuously updated GitHub repository to track the latest advances in this area.