Reinforcement learning (RL) has shown promise in single-turn LLM fine-tuning, but long-horizon agentic reasoning introduces branching interactions and sparse rewards, making RL less effective. The heavy backpropagation-based training stack of RL makes it impractical for larger models, and credit assignment becomes harder with longer trajectories.
In a new paper, researchers from Hugging Face argue that evolution strategies (ES) offer a better alternative for fine-tuning long-horizon agents. ES provides three key advantages: model scalability (full-parameter optimization with minimal inference-level GPU memory), flexibility (easy composition with prompt-space evolution), and long-horizon scalability (trajectory-level parameter attribution without decomposing rewards).
Based on this insight, they propose Agentic ESOpt, a full-parameter agentic fine-tuning framework that supports parameter-context co-evolution. At each step, it samples perturbations around the current parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. A cosine decay schedule for the perturbation scale σ improves the exploration-adaptation trade-off.
On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improved the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performed online prompt-parameter co-evolution, improving its matched baseline in 28 of 36 settings.