Off-policy reinforcement learning (RL) has long relied on stabilizers designed for data-limited replay buffers. However, with the rise of massively parallel simulation, the data regime has shifted, challenging these stabilizers' effectiveness. Researchers at Hugging Face have conducted controlled experiments across eight benchmark families to investigate this issue.
Their findings reveal that stabilizers are data-regime-dependent: parameter normalization helps when replay coverage is narrow but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity.
Based on these insights, the team proposes WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC introduces Sample Weight Decay for efficient exploitation and offers two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training.
In evaluations, WarpSAC improves normalized score-step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It also increases the UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 compared to FlashSAC.
The project page provides an overview, demonstrations, benchmark results, and sim-to-real evaluations, while the GitHub repository contains source code, installation instructions, training scripts, environment integrations, and experiment configurations.