Reinforcement learning (RL) has proven effective for enhancing reasoning in language and vision-language models, but its most successful applications typically rely on ground-truth supervision, such as verifiable rewards. These annotations are expensive to obtain and become increasingly scarce as models surpass human evaluation capabilities. Self-rewarding RL offers an alternative by using a model's own completions as reward signals, yet this approach can reinforce existing biases, reduce response diversity, and lead to training collapse.
In a new paper, Hugging Face researchers introduce Co-RL, a framework that enables unsupervised reasoning through cooperative multi-agent training. Multiple decoupled models, sharing no parameters, are optimized simultaneously using rewards derived from their peers. This peer-based reward mechanism avoids the pitfalls of self-rewarding RL by reducing correlated errors that drive self-reinforcing feedback loops.
The key to Co-RL's success is cohort diversity. By using heterogeneous model families, sizes, and rephrased training samples, the framework maintains behavioral diversity and mitigates training collapse. This diversity consistently improves reasoning performance across both text-only and multimodal domains, outperforming base models and prior label-free approaches while matching or surpassing supervised methods.
Concretely, Co-RL achieves average gains of 3.0-8.6% across seven text-only benchmarks for large language models (LLMs) and 2.3-7.2% across four multimodal benchmarks for vision-language models (VLMs), all without access to ground-truth labels. The code is available on GitHub.