Jiajun Fan

NeurIPS 2025

Adaptive Divergence Regularized Policy Optimization for Fine‑tuning Generative Models

  • UIUC

At a glance

The paper in four results.

Reinforcement learning fine-tuning keeps a generative model near where it started with one fixed penalty, the same for every sample. What if each sample set its own? ADRPO loosens it for samples better than their batch and tightens it for worse ones.

  • Per sample.each sample’s advantage sets its own regularization
  • Plug-and-play.one coefficient in an existing loss, no new networks
  • Across modalities.images, text and audio reasoning

The problem

Fine-tuning holds every sample back equally hard. That forces a trade.

Strong regularization keeps what the model could do but limits how far the reward can rise. Weak regularization lets the reward rise but risks collapse and reward hacking. PPO, GRPO and DPO all pick one strength and use it for every sample.

Strong, fixed

one large β for every sample

Safe, but the reward barely moves.

  • Reaches high reward
  • Keeps its samples varied

Weak, fixed

one small β for every sample

The reward climbs, and the samples collapse into one.

  • Reaches high reward
  • Keeps its samples varied

Adaptive: ADRPO

β0 − A, set by each sample

The reward climbs, and the samples stay varied.

  • Reaches high reward
  • Keeps its samples varied
Three ways to set the regularization, played in turn. Schematic: nothing in it is measured; the measured results follow below.

A good sample and a bad one are held back exactly the same.

The idea Adaptive regularization

Let each sample set its own strength: β0 − A.

A is the sample’s advantage: how much its reward beats the average of its batch. The model may move further toward good samples, and stays close to where it started on poor ones.

One coefficient, set per sample

The advantage is the signal the update already uses. ADRPO reads it a second time to set the regularization.

  • Better than its batch, A > 0Weaker regularization: the model can exploit it.
  • Worse than its batch, A < 0Stronger regularization: the model stays near the original.
  • ClippedA is kept within a fixed range for stable training.

Exploit where the samples are good, stay careful where they are not.

How it works

One coefficient changes, in the loss you already use.

In GRPO, only the fixed β changes: it becomes β0 − A. On flow models, the reward weight also becomes the advantage.

Flow models, with W2

Online fine-tuning of SD3; the divergence compares its velocity field with the pre-trained one.

Fixed: ORW-CFM-W2L = LORW + β · ‖vθ − vref‖2

ADRPOL = LRL + (β0 − A) · ‖vθ − vref‖2

LRL weights each sample by its advantage, so poor samples are pushed away rather than only weighted down.

Language and audio models, with KL

GRPO fine-tuning of an LLM or a multimodal reasoning model; the divergence is the KL to the reference model.

Fixed: GRPOL = LPG + β · DKL(πθ ‖ πref)

ADRPOL = LPG + (β0 − A) · DKL(πθ ‖ πref)

A is GRPO’s own group advantage; β0 equals the fixed β it replaces.

  • no new networks
  • no architectural changes
  • no manual schedule

Nothing new to train, and no schedule to tune.

The evidence Finding 01

A 2B model beats models up to six times its size.

Fine-tuned with ADRPO, SD3 (2B) scores above SANA-1.5 (4.8B) and FLUX.1-Dev (12B) on all six metrics, and keeps the styles, colors and layouts the prompts ask for.

Two samples per model, from the paper

PromptVan Gogh style astronaut

SD3 2B

SD3 2B, sample 1: Van Gogh style astronautSD3 2B, sample 2: Van Gogh style astronaut

SANA-1.5 4.8B

SANA-1.5 4.8B, sample 1: Van Gogh style astronautSANA-1.5 4.8B, sample 2: Van Gogh style astronaut

FLUX.1-Dev 12B

FLUX.1-Dev 12B, sample 1: Van Gogh style astronautFLUX.1-Dev 12B, sample 2: Van Gogh style astronaut

SD3 + ADRPO 2B

SD3 + ADRPO 2B, sample 1: Van Gogh style astronautSD3 + ADRPO 2B, sample 2: Van Gogh style astronaut

Samples from Figures 1 and 5 of the paper. Prompts from DrawBench and RAFT, and compositional prompts written for the paper.

  • SD3 2B
  • SANA-1.5 4.8B
  • FLUX.1-Dev 12B
  • SD3 + ADRPO 2B

Mean over 3 seeds (Table 1). Higher is better on every metric; each panel has its own scale.

How a model is fine-tuned can matter more than how big it is.

The evidence Finding 02

The highest reward, and the diversity stays.

DPO stalls, and ORW-CFM-W2 buys its reward with diversity. ADRPO climbs highest, keeps its samples varied and stays closest to SD3.

Paths: checkpoints during training, one seed, read from Figure 3 of the paper. Dots: final scores, mean over 3 seeds (Table 1).

Two samples per method, all from SD3

PromptA storefront with ‘Diffusion’ written on it.

SD3

SD3, sample 1: A storefront with ‘Diffusion’ written on it.SD3, sample 2: A storefront with ‘Diffusion’ written on it.

+ DPO

+ DPO, sample 1: A storefront with ‘Diffusion’ written on it.+ DPO, sample 2: A storefront with ‘Diffusion’ written on it.

+ ORW-CFM-W2

+ ORW-CFM-W2, sample 1: A storefront with ‘Diffusion’ written on it.+ ORW-CFM-W2, sample 2: A storefront with ‘Diffusion’ written on it.

+ ADRPO

+ ADRPO, sample 1: A storefront with ‘Diffusion’ written on it.+ ADRPO, sample 2: A storefront with ‘Diffusion’ written on it.

Samples from Figure 2 of the paper. ORW-CFM-W2’s two samples often look alike; ADRPO’s get the text, the position and the color right and still differ.

No trade between reward and diversity.

The evidence Finding 03

It escapes a local optimum by itself.

Plugged into GRPO to fine-tune Qwen3 (0.6B), ADRPO first narrows its outputs, then widens them again to climb out. GRPO stays at high entropy and struggles to find higher reward.

  • ADRPO
  • GRPO fixed β
  • Qwen3 0.6B before fine-tuning

Each dot is a checkpoint during training, read from Figure 4 of the paper; the lines are running averages. Reward from the RM-Gemma-2B reward model on held-out prompts; entropy measures how widely the model explores.

It explores more when it is stuck, without being told to.

The evidence Finding 04

A 7B audio model beats Gemini 2.5 Pro and GPT-4o Audio.

Fine-tuned on AVQA and tested on MMAU, Qwen2.5-Omni-7B with ADRPO answers more audio questions right than with GRPO, and more than both commercial models.

Temporal sequenceWhat was the last sound in the sequence?footsteps, dog barking, camera shutter clicking, tapping on glass

GRPOThe last sound in the sequence is a camera shutter click, which is a distinct sound from the other options provided.Camera shutter clicking, wrong
ADRPO… The last sound is a distinct ‘tapping on glass’, which is a clear and unique sound compared to the others.Tapping on glass, correct

Source identificationBased on the given audio, identify the source of the flowing water.Bathtub, River, Fountain, Rain

GRPOThe sound is consistent with a steady flow of water, likely from a man-made source like a bathtub or fountain rather than natural sources like a river or rain.Fountain, wrong
ADRPOThe audio suggests a continuous, steady flow of water, which is characteristic of a bathtub. …Bathtub, correct

Contextual reasoningConsidering the information in the audio, what type of weather condition can be inferred?Windy, Calm, Rainy, Snowy

GRPOThe presence of wind noise and the sound of rain hitting a surface suggest a windy and rainy weather condition.Rainy, wrong
ADRPOThe presence of wind noise suggests a windy condition. The absence of rain or snow sounds indicates it’s not rainy or snowy.Windy, correct

Accuracy from Table 2; reasoning from Table 5 of the paper, both models fine-tuned from Qwen2.5-Omni-7B.

It keeps reasoning where GRPO settles too early.

Why it matters

One rule for images, text and audio.

ADRPO plugs into the reinforcement learning fine-tuning people already run, for continuous and discrete models alike.

Images

Flow matching on SD3, with W2 regularization.

beats 4.8B and 12B models

Text

GRPO on Qwen2 and Qwen3, with KL regularization.

escapes a local optimum

Audio reasoning

GRPO on Qwen2.5-Omni-7B, with KL regularization.

beats Gemini 2.5 Pro

What this opens up

Smaller models that compete

A small model fine-tuned well can stand with much larger ones, and it costs less to run.

Exploration when it pays

The advantage tells the model when to stay close and when to move on, so it can climb out of a local optimum.

A drop-in change

One coefficient becomes a function of the advantage. In GRPO, nothing else changes.

The film

The story, as a short film.

Opens the film page.

Cite

Cite the paper.

Conference on Neural Information Processing Systems 2025 (NeurIPS 2025).

@inproceedings{fan2025adaptive,
  title     = {Adaptive Divergence Regularized Policy Optimization
  for Fine-tuning Generative Models},
  author    = {Jiajun Fan and Tong Wei and Chaoran Cheng and Yuxin Chen and Ge Liu},
  booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems},
  year      = {2025},
  url       = {https://openreview.net/forum?id=aXO0xg0ttW}
}