Jiajun Fan

Fine-tuning Flow Matching Generative Models with Intermediate Feedback

  • 1UIUC
  • 2CMU

At a glance

The paper in four results.

Can an image model learn from feedback on every step of its generation, not only on the finished picture? AC-Flow trains a small critic that scores each intermediate step of Stable Diffusion 3, and keeps it stable while the model learns online.

  • Step-level credit.a critic scores every intermediate step
  • Stable online.it learns from its own samples without collapsing
  • One GPU.a small critic, and each run on a single GPU

The problem

The reward sees only the finished image, and the obvious fix collapses.

A flow model draws an image in many small steps, yet most fine-tuning methods score only the last one. A critic could score every step, but trained naively online, it breaks.

Score only the finished image

what most fine-tuning methods do

Every step gets the same credit, so an early mistake goes uncorrected and a good step goes unrewarded.

Add a critic, naively

value regression, trained online on its own samples

The critic and the model feed each other’s errors: the loss runs away and the images collapse.

Schematic. The loss scale and the three images come from the paper’s plain actor-critic run (Fig. 2).

What is missing: credit for each step, from a critic that stays stable.

The idea Step-level credit

Keep the update.
Let a critic weight each step.

Reward-weighted fine-tuning scales a flow-matching update by how good the final image was. AC-Flow replaces that one number with a weight for every step, set by the reward minus what the critic expected from that step.

Schematic, after the paper’s Fig. 1. Bars: what sets each step’s weight. In AC-Flow, a bar above the line makes a step count for more, below it for less.

Steps that beat the critic’s expectation count for more; steps that fall short count for less.

How it works

One critic, three safeguards.

The critic is small: a few convolution and MLP layers beside Stable Diffusion 3. Three safeguards make its advice safe to follow online, and a Wasserstein-2 penalty keeps the model close to where it started.

  • no image dataset
  • no likelihood
  • no hyperparameter tuning
  • no GPU cluster

A critic it can trust, and a model kept close to where it started.

The evidence Finding 01

Its critic trains without blowing up.

Plain actor-critic training on Stable Diffusion 3 never settles. Each safeguard pulls the actor loss further down; with all three, both losses come down early and stay down.

plain actor-critic+ reward shaping+ advantage clippingAC-Flow, all three

Actor losslog scale

Value losslog scale

The ablation of the paper’s Fig. 2, redrawn from its curves. Shaded: the critic’s warm-up, the first 500 steps.

Reward shaping tames the scale, clipping tames the spikes, and the warm-up lets the critic mature before it steers.

The evidence Finding 02

It keeps improving where plain actor-critic collapses.

One prompt, “A dog in the sky”. Without the safeguards the reward peaks early and slides back. AC-Flow keeps climbing.

Plain actor-critic “A dog in the sky”

A dog's face filling the frame against blue skyAlmost the same close-up of a dog's faceAlmost the same close-up of a dog's face, distortedAlmost the same close-up of a dog's faceAlmost the same close-up of a dog's faceAlmost the same close-up of a dog's face

Distorted, and nearly the same close-up every time.

AC-Flow “A dog in the sky”

A dark dog standing on clouds under a blue skyA dog lying on a bank of cloudsA brown and white dog standing above the cloudsA small dog sitting on cloudsA dog's head and shoulders above the cloudsA dog running across the sky

Coherent dogs, in the sky, and varied.

Reward curves from the paper’s Fig. 4, redrawn with its shaded bands; images from its Fig. 2.

With a critic it can trust, more training keeps paying off.

The evidence Finding 03

It aligns best, even by judges it never saw.

Every method was fine-tuned on the same reward, CLIP score. AC-Flow scores highest on it, and on two human-preference models that played no part in its training.

DrawBench prompts on Stable Diffusion 3 (SD3, the base model, dashed). Mean and standard error over three seeds, from the paper’s Table 1.

How much variety it keeps

Diversity: the mean distance between CLIP embeddings of images for the same prompt (paper Table 1). The Wasserstein-2 penalty raises alignment and diversity together.

It learns what the prompt asks for, not a trick that pleases one reward model.

The evidence Finding 04

It gets hard prompts right.

Where each object goes, which color belongs to which object, what the sign says. The same prompts, five methods, two samples each.

Samples from the paper’s Fig. 3

Prompt“A banana on the left of an apple.”

SD3SD3: a bunch of bananas with a red apple to its rightSD3: bananas with a green apple in front
Diffusion-DPODiffusion-DPO: a banana leaning on a red appleDiffusion-DPO: bananas behind a green apple
RAFTRAFT: a small curled banana and an apple on a table, washed outRAFT: almost the same washed-out scene
RWR+W2RWR+W2: a banana arching over an appleRWR+W2: a banana lying across, an apple to the right
AC-FlowAC-Flow: a banana on the left of a red appleAC-Flow: a banana standing to the left of a red apple
SD3SD3: a green backpack with a red and a dark red appleSD3: a green backpack with a dark red apple and a green apple
Diffusion-DPODiffusion-DPO: a green backpack between dark and red applesDiffusion-DPO: a green backpack with red and black apples
RAFTRAFT: a red apple and a small green backpack on a tableRAFT: almost the same scene
RWR+W2RWR+W2: a black apple and a green apple beside a green backpackRWR+W2: a dark apple and a red apple beside a green backpack
AC-FlowAC-Flow: a dark apple beside a green backpackAC-Flow: a black apple and a green backpack
SD3SD3: two green cups and a blue phoneSD3: a green cup and a dark phone with a blue case
Diffusion-DPODiffusion-DPO: a green cup and a blue phoneDiffusion-DPO: a green cup and a blue phone
RAFTRAFT: a green cup and a blue phone on a green trayRAFT: a green cup and a blue phone on a white tray
RWR+W2RWR+W2: two green cups and a blue phoneRWR+W2: green cups and a blue phone
AC-FlowAC-Flow: one green cup and one blue phoneAC-Flow: one green cup and one blue phone
SD3SD3: a storefront with a sign reading Text to imageSD3: a storefront with the words broken up across its doors
Diffusion-DPODiffusion-DPO: a storefront under a red awning, its sign reading Text to ImageDiffusion-DPO: a storefront with a sign reading Text to Image
RAFTRAFT: a storefront whose signs read Text Image and Textt to ImageRAFT: a storefront with two signs reading Text to Image
RWR+W2RWR+W2: a storefront with a sign reading textt to imageRWR+W2: a storefront with a sign reading Text to Image
AC-FlowAC-Flow: a storefront with a clear sign reading Text to ImageAC-Flow: a storefront with a clear sign reading Text to Image

“A sign that says ‘Hello World’.”

SD3SD3: a sign reading helllo WORLD, misspelledSD3: wall letters reading HELLLO WORLD, misspelled
AC-FlowAC-Flow: a lit sign reading HELLO WORLDAC-Flow: a sign reading Hello World

“A blue coloured pizza.”

SD3SD3: an ordinary pizza on a blue plateSD3: an ordinary pizza on a blue background
AC-FlowAC-Flow: a pizza with a blue topAC-Flow: a pizza with blue sauce

“An elephant under the sea.”

SD3SD3: an elephant on a coral sea floorSD3: an elephant under water with an orange creature above
AC-FlowAC-Flow: an elephant standing among coral under the seaAC-Flow: an elephant facing forward under the sea

“In late afternoon in January in New England, a man stands in the shadow of a maple tree”

SD3SD3: a man standing in a snowy park near treesSD3: a man on a snowy shore beside a tree
AC-FlowAC-Flow: a man standing under a maple tree with red leaves in the snowAC-Flow: a man under a red maple tree in a snowy field

More prompts against the base model, from the paper’s Fig. 5.

Feedback on every step gives finer control over what ends up in the image.

Why it matters Finding 05

Step-level feedback, for about the cost of scoring the end.

The critic adds little. Next to reward-weighted fine-tuning, a run takes about two hours and two gigabytes more, and every run fits on one GPU.

Run timehours per run

Peak GPU memorygigabytes, on one 48 GB GPU

One NVIDIA RTX A6000 per run, from the paper’s Table 2.

What this opens up

Fine-tuning within reach

A small critic and a light setup let online fine-tuning of Stable Diffusion 3 run on one widely available GPU.

Models that learn from their own samples

Stable online training on self-generated data points to generators that keep improving through their own outputs.

Any goal you can score

Swap in another reward, such as one for safety, and the same loop steers the model toward it.

Part of a series on fine-tuning flow models online: ORW-CFM-W2 and ADRPO.

The film

The story, as a short film.

Opens the film page.

Cite

Cite the paper.

@misc{fan2025finetuningflowmatchinggenerative,
  title={Fine-tuning Flow Matching Generative Models with Intermediate Feedback},
  author={Jiajun Fan and Chaoran Cheng and Shuaike Shen and Xiangxin Zhou and Ge Liu},
  year={2025},
  eprint={2510.18072},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2510.18072}
}