Hugging Face researchers have introduced FlowBalance, a novel method for improving reasoning models through verifier-grounded self-improvement. The approach addresses a critical fragility in the inner loop of self-improvement, where models learn from their own on-policy experience.
Traditional methods rely on terminal verifiers for reliable but sparse supervision, while dense same-model guidance can reinforce false confidence or over-concentrate learning on a narrow solution mode. FlowBalance tackles this by learning a normalized distribution over complete responses, using a frozen training-time view of the policy to compute token-level log-probability gains, aggregated into a trajectory-level self-guidance score.
The key innovation is calibrating this self-guidance score with a verifier-derived group advantage. Guidance is retained on positive-advantage trajectories, reversed on negative-advantage ones, and disabled when no outcome preference exists. This calibration prevents the model from reinforcing verifier-wrong traces, a known failure mode of other methods like OPSD.
In experiments on mathematical reasoning, FlowBalance improved average performance over FlowRL on both Qwen3-4B and Qwen3-8B models. It also enhanced training speed and stability, avoided response-length collapse, and exhibited higher correct-strategy diversity in a controlled AIME24 diagnostic.