Hugging Face researchers have introduced ClawGym II, a unified black-box reinforcement learning (RL) framework designed to optimize general agents through complex harnesses. The framework addresses the challenges of scaling RL to long-horizon agent tasks by decoupling policy optimization from opaque harness execution.
The approach builds a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes, enabling large-scale concurrent rollouts. A serving proxy at the model boundary captures model calls, which are organized into prefix trees to reconstruct multi-turn trajectories. Both critic-based PPO and critic-free GRPO are adapted to optimize over these tree structures, while maintaining training-inference consistency.
Mix-harness training is a key innovation, allowing a single model to be jointly optimized by heterogeneous harnesses. Experiments with Qwen3-30A3B show that black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, with stability over 200-400 optimization steps. Consistent gains are also observed on more challenging tasks like JobBench and OfficeQA.
The framework demonstrates effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.