CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Published in arXiv preprint, 2026
Recommended citation: Licheng Pan, Haocheng Yang, Yuzhou Chen, Yichen Sun, Yuan Lu, Yunfei Teng, Jiajun Fan, Guangyi Liu, Zhixuan Chu, Hao Wang. "CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF." arXiv preprint arXiv:2610.21138, 2026. https://arxiv.org/abs/2610.21138
CleanRM trains reward models from noisy preference feedback with the help of a few clean labels: optimal transport calibrates a noise ratio for each sample, and a noise-corrected surrogate loss makes the objective unbiased with respect to the clean risk. It beats denoising baselines and gives better downstream RLHF alignment.