NeurIPS 2025
Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
In brief
What the paper does.
We propose ADRPO, which dynamically adjusts divergence regularization strength based on advantage estimates — reducing regularization for high-value samples while applying stronger constraints to poor samples. Enables a 2B SD3 model to surpass 4.8B/12B models; also generalizes to LLMs and multimodal reasoning.
Cite
Cite the paper.
@inproceedings{fan2025adaptive,
title = {Adaptive Divergence Regularized Policy Optimization
for Fine-tuning Generative Models},
author = {Jiajun Fan and Tong Wei and Chaoran Cheng and Yuxin Chen and Ge Liu},
booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year = {2025},
url = {https://openreview.net/forum?id=aXO0xg0ttW}
}