A new research paper titled "Sharpening Tax in Post-Training" challenges a prevailing hypothesis about reinforcement learning (RL) post-training of large language models (LLMs). The hypothesis suggests that RL post-training merely sharpens behaviors already present in a base model, boosting single-shot accuracy at the expense of solution coverage. While this trade-off has been documented in math and coding tasks, the authors argue it may not necessarily apply to agentic tasks, where multi-turn tool use and interaction could demand capabilities acquired during post-training.
Surprisingly, the researchers find that pre-trained LLMs equipped with a light inference harness can serve as capable agents. Despite much lower pass@1 accuracy, these base models often outperform their post-trained counterparts in solution coverage (pass@K) when given a sufficient test-time budget. The team analyzes the underlying mechanism, showing that post-training pushes tasks toward two extremes: either always solved or never solved. This improves sampling efficiency and consistency but comes at the cost of solution coverage.
To quantify this cost, the authors introduce Sharpening Tax, a diagnostic metric that measures the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases total), the tax appears in most settings, can be estimated from a few rollouts, and correlates well with other metrics.
Finally, the paper presents posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt based on estimated difficulty. When applied during RL training in two agentic environments, PTGS incurs a smaller tax than a fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.