Tool-calling agents generate heterogeneous outputs, mixing structured tool invocations with natural-language summaries. According to a new paper, this heterogeneity exposes a structural failure mode in standard on-policy reinforcement learning: algorithms such as GRPO broadcast a single trajectory-level scalar advantage to every token, so gradient noise from summary generation leaks into tool-decision tokens. The authors call this cross-segment credit misattribution.
To address it, the paper proposes SLCA-GRPO, built around Segment-Locked Credit Assignment (SLCA). The method decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), it routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination within each policy update.
The authors also introduce the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure, enabling scalable exploration without costly real APIs and supporting stable training.
On a 7B backbone, SLCA-GRPO is reported to accelerate convergence and outperform standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on τ²-Bench under the same training budgets. The paper states this comes with higher accuracy and reduced tool redundancy and costs.
The authors further report that across three Qwen backbones and both in- and out-of-distribution tool-calling benchmarks, the approach improves task success while reducing tool turns.