Reinforcement learning (RL) search agents typically generate free-form natural-language queries to retrieve evidence, but researchers at Hugging Face have identified a critical flaw in this approach: retrieval-equivalence collapse. During training, agents often produce diverse query strings that lead to nearly identical evidence sets, leaving little contrast for optimization and reducing the effectiveness of group-relative credit assignment.
To address this, the team introduces Harness-G, a graph-structured retrieval framework that redesigns the policy-environment interface. Instead of generating free-form queries, the policy selects from a finite menu of typed, verifiable actions—such as choosing an evidence sentence or entity, or deciding to answer. The environment constructs the menu, tracks retrieval state, and validates and executes each choice, reducing linguistic aliasing and making alternatives directly comparable.
Building on this interface, the authors propose Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to earlier actions that enabled them. This provides denser, more structured credit signals for training.
Across six question-answering benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales (1.5B and 3B), outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B. The work highlights the importance of properly formulating retrieval at the policy-environment interface for effective RL training.