📝 Preprint 2025 · Under Review

Fine-tuning Flow Matching Generative Models with Intermediate Feedback

AC-Flow: Robust actor-critic framework for flow matching — stable intermediate value learning without collapse

Jiajun Fan ¹, Chaoran Cheng ¹, Shuaike Shen ¹, Xiangxin Zhou ¹, Ge Liu ¹

¹University of Illinois Urbana-Champaign

TL;DR — Existing RLHF methods for flow matching only use outcome rewards (ORW-CFM-W2), suffering from credit assignment problems. AC-Flow introduces a full actor-critic framework with intermediate feedback — reward shaping + dual-stability + generalized critic weighting — achieving SOTA text-to-image alignment on SD3 without degrading diversity or stability.

🔧 Three Key Innovations

Reward Shaping

Provides well-normalized learning signals for stable intermediate value learning and gradient control — enabling the critic to reason about multi-step trajectories.

Dual-Stability Mechanism

Combines advantage clipping (prevents destructive policy updates) with a critic warm-up phase (lets critic mature before guiding the actor).

Generalized Critic Weighting

Extends reward-weighted methods while preserving model diversity via Wasserstein regularization — compatible with ORW-CFM-W2 as a special case.

🧩 Why Intermediate Rewards Matter

Standard RLHF for flow matching gives feedback only at the end of the denoising chain — a sparse signal over many steps. AC-Flow instead assigns intermediate rewards at each timestep via an actor-critic formulation:

Denser gradients → more stable learning, fewer catastrophic updates
Critic as a value estimator → predicts future reward quality at each denoising step
Decoupled stability → advantage clipping handles actor; critic warm-up handles the critic

The result: fine-tuning that generalizes beyond direct reward optimization — improvements transfer to unseen LLM and audio tasks (not just image generation).

🔄 AC-Flow vs ORW-CFM-W2

ORW-CFM-W2 (ICLR 2025)

Outcome reward only
No intermediate value learning
Credit assignment challenge
First online RLHF for flow matching

AC-Flow (This Work)

Intermediate feedback + actor-critic
Stable value learning via reward shaping
Dual-stability prevents collapse
SOTA on SD3 with even less data

AC-Flow generalizes ORW-CFM-W2 — the critic weighting scheme subsumes reward-weighted methods as a special case.

📊 Key Claims

⚙️ Actor-Critic

Step-level rewards via
intermediate feedback

🛡️ Collapse-Free

Wasserstein reg +
dual-stability mechanism

🎯 SD3 Fine-Tuning

Stable fine-tuning without
mode collapse

🔗 Related Work in This Series

AC-Flow builds on the online RLHF series for generative models:

ORW-CFM-W2 (ICLR 2025) — First online RLHF for flow matching
ADRPO (NeurIPS 2025) — Adaptive sample-level regularization for RLHF
AC-Flow (This work) — Actor-critic with intermediate step-level rewards

📖 Cite This Paper

@misc{fan2025finetuningflowmatchinggenerative,
  title={Fine-tuning Flow Matching Generative Models with Intermediate Feedback},
  author={Jiajun Fan and Chaoran Cheng and Shuaike Shen and Xiangxin Zhou and Ge Liu},
  year={2025},
  eprint={2510.18072},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2510.18072}
}