Tool-Integrated Reasoning (TIR) allows large language models (LLMs) to tackle complex tasks by interacting with external tools iteratively. However, existing reinforcement learning (RL) methods often rely on trajectory-level supervision, which limits fine-grained credit assignment in long-horizon scenarios. On-policy self-distillation offers denser signals, but typically derives them from ground-truth answers or retrieved skills, which may not align with the actual states visited by the agent.
To address these limitations, researchers at Hugging Face propose TurnSight, a turn-level hindsight self-distillation framework. TurnSight derives supervision directly from execution-conditioned hindsight—that is, from the tool execution outcomes the agent actually encounters. It constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement.
The selected hindsight signal is then normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. This allows the model to distinguish between useful, redundant, and harmful tool-use decisions across long trajectories.
Extensive experiments on three benchmarks demonstrate TurnSight's effectiveness, robustness, and generalization ability across diverse multi-turn TIR scenarios. The code, datasets, and model checkpoints are fully open-sourced on GitHub and Hugging Face.