Jiajun Fan

NeurIPS 2025

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models

In brief

What the paper does.

We propose ADRPO, which dynamically adjusts divergence regularization strength based on advantage estimates — reducing regularization for high-value samples while applying stronger constraints to poor samples. Enables a 2B SD3 model to surpass 4.8B/12B models; also generalizes to LLMs and multimodal reasoning.

Published in
The Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
Paper
openreview.net/forum?id=aXO0xg0ttW
Project page
jiajunfan.com/projects/adrpo/

Cite

Cite the paper.

@inproceedings{fan2025adaptive,
  title     = {Adaptive Divergence Regularized Policy Optimization
  for Fine-tuning Generative Models},
  author    = {Jiajun Fan and Tong Wei and Chaoran Cheng and Yuxin Chen and Ge Liu},
  booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems},
  year      = {2025},
  url       = {https://openreview.net/forum?id=aXO0xg0ttW}
}