ADRPO — Adaptive Divergence for Generative Models

TL;DR — Fixed divergence regularization in RLHF forces a dilemma: strong regularization preserves diversity but limits alignment; weak regularization aligns better but risks collapse. ADRPO adapts the regularization strength per sample from the advantage estimate — high-value samples explore freely, poor samples stay constrained. One line of math; works with W2 (flow matching) and KL (LLMs).

Key idea: not all samples deserve equal regularization

Make the regularization strength a function of sample quality

High-advantage samples (clearly better than the batch average) deserve less regularization so the model can exploit them; low-advantage samples need more regularization to prevent harmful updates. The effective coefficient becomes advantage-dependent:

β_eff = β₀ − A

where A is the sample's advantage and β₀ a baseline coefficient. No new networks, no manual KL schedule — a one-line change to existing RLHF objectives.

Fixed regularization (DPO / ORW-CFM-W2)

Same strength for every sample
Strong → conservative, slow
Weak → reward hacking / collapse
Manual coefficient tuning

ADRPO (ours)

Adaptive from advantage estimates
High-value samples explore freely
Poor samples kept on a short leash
Plug-and-play — no extra networks

Results on SD3 text-to-image

SD3 surpasses 4.8B & 12B models

32.6

reward — highest of all methods

> DPO

higher reward & diversity

extra networks (plug-and-play)

A 2B SD3 model fine-tuned with ADRPO matches or outperforms FLUX.1-Dev (12B) and SANA-1.5 (4.8B) on attribute binding, style rendering, and compositional control — while reaching both the highest reward and the highest diversity.

Figure 1. Final reward vs. diversity on SD3. ADRPO reaches the top-right — the highest reward and the highest diversity — while ORW-CFM-W2 trades away diversity for reward and DPO lands in between.

Figure 2. Reward vs. W2 divergence from the pre-trained model (flow matching uses W2, not KL). ADRPO attains the highest reward while staying closest to the reference — the smallest divergence — whereas ORW-CFM-W2 needs far larger divergence for less reward.

Generalizes across modalities

ADRPO is a universal plug-in — the same advantage-adaptive principle works for flow-matching, text LLMs, and audio reasoning:

Flow / diffusion (SD3) · W2 Text LLMs (GRPO) · KL Audio reasoning LLMs

In LLM fine-tuning, ADRPO shows an emergent ability to escape local optima: when the model stalls, the adaptive coefficient globally raises regularization and re-opens exploration, then relaxes once a promising direction appears.

Figure 3. Reward over training on Qwen2 (GRPO vs. ADRPO). ADRPO escapes the local optimum GRPO stays trapped in, climbing to roughly 3× the reward — the adaptive coefficient re-opens exploration whenever the model stalls.

Publication journey

May 2025

Submitted to NeurIPS 2025

The Thirty-ninth Annual Conference on Neural Information Processing Systems.

Sep 2025

Accepted at NeurIPS 2025 (Poster)

OpenReview.

Oct 2025

arXiv preprint released

arXiv:2510.18053.

Dec 2025

Presented · San Diego, CA

NeurIPS 2025.

BibTeX

@inproceedings{fan2025adaptive,
  title     = {Adaptive Divergence Regularized Policy Optimization
               for Fine-tuning Generative Models},
  author    = {Jiajun Fan and Tong Wei and Chaoran Cheng and Yuxin Chen and Ge Liu},
  booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems},
  year      = {2025},
  url       = {https://openreview.net/forum?id=aXO0xg0ttW}
}

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models