One Subtraction Ends the Exploration–Exploitation Trade‑off
RL fine-tuning of a generative model runs on one dial. Turn it up and the model keeps its diversity but stops improving; turn it down and it chases reward until it collapses into template output. The dial is a single coefficient applied identically to every sample. This note is about what happens when you stop treating it as a constant.
βtot = β0 − A+0.00 = 1.00
Average sample. The advantage is near zero, so the coefficient stays at its baseline 1.00 and the sample barely moves the policy.
One subtraction, and the dilemma dissolves. Every RL fine-tuning method before this one picks a single β and applies it to every sample: raise it and the model is protected but cannot improve, lower it and it improves until it collapses. ADRPO — Adaptive Divergence Regularized Policy Optimization — makes β a function of the sample — Equation 4 in the paper is literally βtot = β0 − A. Good samples get freedom, bad samples get held down, and nobody has to tune the trade-off. β0 = 1 with A clipped to [−1, 1] is the paper’s text-to-image setting.
The short version
Six things one subtraction bought
Each is stated with the number that carries it, and shown in full further down.
Strong regularisation protects the pre-trained model but caps the reward. Weak regularisation chases reward and invites collapse or hacking. Every sample gets the same β, whether it deserves it or not.
ADRPO sets β = β0 − A. High-advantage samples are regularised less and exploited harder; low-advantage samples are pulled back toward the reference model. One term, no extra networks, no architecture change.
Reward-weighting can only give a bad sample a small positive push. An advantage-weighted objective flips the gradient sign and pushes the policy away from that sample — actively suppressing poor generations instead of politely ignoring them.
Fine-tuned SD3 at 2B parameters outscores FLUX.1-Dev (12B) and SANA-1.5 (4.8B) on ClipScore, aesthetics and human preference. Adaptive regularisation bought more than a 6× parameter increase would have.
Every competing method trades one for the other. ADRPO finishes at 5.13 diversity against the base model’s 5.08 — level to within seed noise, while posting the highest alignment score in the table.
On Qwen3, ADRPO escapes a local optimum by deliberately raising entropy and converges at 5× GRPO’s reward. On MMAU a 7B model reaches 76.0, past Gemini 2.5 Pro and GPT-4o Audio.
01 — The dilemma
The coefficient that has to be two things at once
Every mainstream RL fine-tuning objective — PPO, GRPO, DPO, and W2-regularised flow matching — carries a divergence penalty scaled by a fixed β. That single number is asked to do two opposing jobs simultaneously: keep the policy near the pre-trained model so it does not forget or collapse, and get out of the way so the policy can actually improve.
The bind is that the right answer differs per sample. A generation the reward model scores highly is a direction worth committing to; holding it back is pure loss. A poor generation is exactly where you want the reference model’s pull to be strongest. A constant cannot express that, so practitioners tune β to a compromise that is wrong for both cases.
02 — The fix
Subtract the advantage
ADRPO replaces the constant with β = β0 − A, where A is the advantage estimate already being computed for the policy gradient. Nothing is added to the model, no second network is trained, and the objective stays a drop-in for existing methods. What changes is that the constraint now varies inversely with sample quality.
03 — Text-to-image
A 2B model past a 12B one
ADRPO was applied to SD3 (2B parameters) on DrawBench prompts with CLIP score as the reward, against offline DPO, reward-ranked RAFT, fixed-β ORW-CFM-W2, and two much larger models that were never RL-tuned at all. It leads on task metrics, image quality and human preference at the same time.
Six metrics, one table, nothing cherry-picked. The red line is the un-tuned SD3 the fine-tuned rows all start from. Click through the metrics and the shape of the problem shows: RAFT falls behind the base model on four of the six, ORW-CFM-W2 buys alignment and pays in diversity, and ADRPO is the only row that is at or above the base model on every one — and it leads all six outright. Two of the rows are larger models — 6× and 2.4× its parameter count.
04 — The Pareto front
Reward went up. Diversity did not go down.
This is the result that is hard to get by tuning a constant. Reward-ranked fine-tuning buys almost no alignment at all (29.35 against the base model’s 29.27) and still flattens the output distribution to pay for it — diversity falls from 5.08 to 1.85, a textbook collapse. Fixed-β W2 regularisation is gentler but still pays 3.86. ADRPO finishes at 5.13 against the base model’s 5.08 — unchanged to within the reported seed noise (±0.47 and ±0.52) — while posting the highest alignment score in the table. It is the only row that buys alignment without paying for it.
05 — Language models
An emergent willingness to explore
The same objective drops into GRPO for LLM fine-tuning by making the KL coefficient advantage-dependent. Tracked in reward–entropy space on Qwen2 (0.5B) and Qwen3 (0.6B), with RM-Gemma-2B supplying the reward, ADRPO and GRPO take visibly different paths. GRPO holds high entropy throughout and moves sideways — lots of exploration, little reward found. ADRPO first tightens into a low-entropy region, then deliberately raises entropy again to break out of the local optimum it landed in, and converges at 5× GRPO’s final reward.
Nobody designed that behaviour. It falls out of the coefficient: once a region stops producing advantage, regularisation rises, the policy loosens, and exploration resumes on its own. The absence of that mechanism is why GRPO’s later checkpoints often score worse than earlier ones while ADRPO improves monotonically — no early stopping required.
06 — Audio reasoning
And it holds in a third modality
Continuous flow matching and discrete token generation are different enough that a shared mechanism is worth testing on a third case. Qwen2.5-Omni-7B was fine-tuned on AVQA with verifiable rewards and evaluated on MMAU.
07 — Robustness
The one hyperparameter it adds barely matters
Adaptive regularisation introduces a clipping range for the advantage. If performance were delicately balanced on it, the method would have traded one tuning problem for another. Sweeping it over a 4× span moves the total by less than half a point, and every setting still beats fixed-β GRPO.
08 — What it means
The trade-off was never the problem. Treating it as constant was.
Exploration versus exploitation is usually framed as something you resolve before training by picking a coefficient, and live with afterwards. The result here is that the information needed to resolve it properly is already sitting in the training loop: the advantage estimate says, for this particular sample, whether the policy has found something worth committing to. Subtracting it turns a global compromise into a local decision.
What makes the result more than a tuning trick is the range it survives. The same one-term change holds across continuous flow matching with a Wasserstein penalty, discrete LLM generation with a KL penalty, and multi-modal audio reasoning — three architectures, two divergence measures, one subtraction, and a 2B model that outperforms a 12B one.