Reinforcement Learning with Verifiable Rewards (RLVR) has been a key driver behind recent advances in reasoning-focused large language models (LLMs), enabling large-scale optimization. However, its use has been largely confined to domains like mathematics and coding, where correctness can be deterministically checked. Open-ended tasks, by contrast, often depend on human preferences, reward models, or LLM-based judges, which introduce evaluation bias, judge capability bottlenecks, and extra inference costs.
Drawing on self-supervised learning principles, researchers from Hugging Face propose RLSVR (Reinforcement Learning with Self-Verifiable Rewards), a task-transformation training paradigm that extends RLVR to open-ended tasks. RLSVR converts open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals, eliminating the need for external judges.
The method is instantiated with SpyRL, a multi-agent self-play environment inspired by the game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Since the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely tied to output quality.
Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code are available at github.com/wangqinsi1/SpyRL.