A recent paper from Hugging Face researchers investigates the optimization behavior of Evolution Strategies (ES) as a post-training paradigm for large language models (LLMs). The study compares ES with Group Relative Policy Optimization (GRPO), a mainstream method, and identifies a distinct advantage for ES in terms of reasoning coverage.
Theoretically, the authors show that verifier-projected Jensen-Shannon diversity across the ES population correlates with higher Pass@K performance. Empirically, while GRPO exhibits entropy collapse, ES improves Pass@1 and achieves higher Pass@K, indicating that ES better exploits the reasoning capabilities of pretrained LLMs.
To leverage both methods, the researchers propose a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. This hybrid approach could offer a more balanced optimization for reasoning tasks.
The paper also reveals that despite substantial whole-model parameter drift, ES's performance gains are driven by a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement does not necessarily lead to widespread functional change or catastrophic forgetting, as confirmed by held-out evaluations.
Finally, the study examines hyperparameter design, showing that ES requires a smaller population size when applied to larger LLMs. These findings position ES as a distinct reasoning post-training paradigm rather than a mere memory-efficient alternative to GRPO.