Standard supervised fine-tuning (SFT) for agents applies loss only to the action tokens the agent authors, treating environment observations as context rather than prediction targets. A new paper asks whether this convention is the best starting point for subsequent reinforcement learning, and proposes an alternative called ActObs.
ActObs additionally supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations themselves, learning to predict them pushes the policy to model the consequences of its actions — without adding data, parameters, sequence tokens, or forward passes.
According to the paper, both approaches perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The benefit also extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), a setting whose tasks are unseen during both SFT and RL.
The authors report that ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Their analysis traces the difference back to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision is said to prevent this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.