Reinforcement learning with verifiable rewards (RLVR) typically converts an agent's experience into learning signals through scalar outcome rewards after interaction. But in group-relative objectives such as GRPO, that signal disappears whenever every rollout in a group receives the same reward — even though the trajectories themselves may contain useful information about what the task requires and how the agent fails.
A new paper on Hugging Face, Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight, asks a complementary question: can hindsight teach an agent what it could have anticipated before acting? The authors introduce prospective learning, which uses post-hoc experience to supervise foresight predictions made from the pre-interaction view, and instantiate it as Self-Retrospection Distillation (SRD).
The intuition is that a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided. SRD distills this privileged hindsight into trajectory-blind foresight from the same policy, aligning token-level distributions between a foresight prediction and a stop-gradient copy of the policy that has seen the completed trajectory. Foresight serves only as a training target and need not be generated at inference time.
According to the authors, SRD is a lightweight auxiliary loss that plugs into GRPO, OPSD, or RLSD. Across 10 tool-integrated reasoning and long-horizon agentic tasks — spanning math, code, search, ALFWorld and WebShop — SRD complements RLVR and self-distillation baselines with gains of up to 24.2 percentage points. Its advantage is especially pronounced when reward contrast is scarce: 37–98% of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories.
The paper highlights a striking 2B-scale result: where 98% of groups are all-failure, RLVR training ends at 0.0% success, while adding SRD reaches 60.6% under the same rollout budget. The authors also report that SRD stabilizes pure self-distillation — 9B OPSD falls below the untrained model on HotpotQA, 2Wiki and LCB-v6, and adding SRD lifts all three back above it — and that gains hold under held-out formats and much longer horizons such as BrowseComp-Plus with 10×+ tool calls.
The authors frame the broader takeaway as evidence that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction. They invite discussion on other kinds of foresight worth distilling beyond pitfalls and knowledge.