Jiajun Fan

ICLR 2025

Online Reward‑Weighted Fine‑Tuning of Flow Matching with Wasserstein Regularization

  • 1UIUC
  • 2CMU
  • 3USC

At a glance

The paper in four results.

Can a flow-matching image model learn from a reward alone, online, from its own samples, without collapsing? Yes: weight its own samples by their reward, and hold it near the base model with a Wasserstein‑2 bound it can compute.

  • Online.it learns from what it just generated
  • Likelihood-free.the bound needs only two vector fields
  • Any reward.a classifier, a file size, CLIP and more

The problem

Reward fine-tuning works for diffusion models. For flow matching, three things stand in the way.

Policy gradients

The likelihood is too costly.

Methods such as DDPO need each sample's likelihood. For a continuous flow, that means integrating the model's divergence along its whole path.

Offline re-weighting

A fixed dataset stops short.

Re-weighting data collected in advance cannot reach the best policy: the online–offline gap. DPO also needs pairs and filtered data.

Online, unchecked

Learning online collapses.

A model that keeps training on its own best samples narrows until it makes one image: the over-optimization seen in diffusion models.

What is missing: an online method with a limit that a flow model can actually compute.

The idea Lemma 1

Learning online reaches the best reward. Left unchecked, it collapses.

We prove that re-weighting its own samples round after round sharpens the model toward the highest reward, until it makes the same image every time. The usual limit, a KL penalty, needs the very likelihood a flow cannot afford.

Schematic of the paper's analysis (Theorem 2, Lemma 1 and Theorem 5), not measured. Bars: share of samples in each of five modes; dashed: the base model.

So keep the online loop, and add a limit the model can compute.

How it works

Learn from its own samples, within reach of where it started.

Each round it weights its own samples by their reward and trains on them. A Wasserstein‑2 bound, computed from the two models' vector fields, keeps it near the base model.

  • no likelihoods
  • no reward gradients
  • no filtered datasets
  • no human-collected data
Reward-weighted flow matching

w(x1) · ‖vθ(t, x) − ut(x | x1)‖2

Trains on its own samples, each weighted by its reward. Online, the weights compound round after round.

The W2 bound, Theorem 3

W22 ≤ e2L ∫01 E ‖vft − vref‖2 ds

Bounds how far it drifts from the base model with only the two vector fields, no likelihood. It enters the loss with weight α.

τ sets how fast it converges; α sets where it settles.

The evidence Finding 01

Online, it reaches the optimum.

Rewarded for even digits, a model trained on its own samples reaches the best possible reward. Re-weighting a fixed dataset stops short.

Fine-tuning an MNIST flow model toward even digits reward = p(even) − p(odd), no W2 term (α = 0)

Online, τ = 10Online, τ = 0.01Offline, RW-CFMNo reward weight, τ = 0Optimal

What each run generates

The paper's own samples, one grid per run. The larger τ, the greedier it gets.

An 8 by 8 grid of handwritten digits of every kind, odd and even
τ = 0every digit, like the pre-trained model
An 8 by 8 grid of even digits: zeros, twos, fours, sixes and eights
τ = 0.01even digits
An 8 by 8 grid that is almost all sixes
τ = 10even digits, less variety

Paper, Fig. 2. Curves traced from the paper's figure, without its shaded spread.

Learning from what it just generated takes it where no fixed dataset reaches.

The evidence Finding 02

A W2 bound stops the collapse.

Rewarded for cats, dogs or birds, the model without the bound draws the same picture over and over. With it, the samples stay varied at a similar reward.

“an image of cat”

An 8 by 8 grid of the same pale blurry shape repeated in every cell
Without W2α = 0
An 8 by 8 grid of small, different cat images
With W2α = 1

“an image of dog”

An 8 by 8 grid of one identical dark image repeated in every cell
Without W2α = 0
An 8 by 8 grid of small, different dog images
With W2α = 1

“an image of bird”

An 8 by 8 grid of one identical green and grey image repeated in every cell
Without W2α = 0
An 8 by 8 grid of small, different bird images
With W2α = 1

The cat run, step by step W2 distance estimated by its upper bound

Paper, Figs. 11, 13 and 14: an unconditional CIFAR-10 flow model with a CLIP reward, τ = 1. Curves traced from Fig. 11; dashed: the pre-trained model.

Without a limit, it drifts ever further from where it started; with the bound, it stays close.

The evidence Finding 03

One knob sets the balance.

Raise α and the model stays closer to the base model, giving up some reward in exchange. Each point on the curves is fine‑tuned separately, at its own α.

Making CIFAR-10 images easy to compress reward from the JPEG file size, τ = 1

Image compression, CIFAR-10 τ = 1

An 8 by 8 grid of varied CIFAR-10 style images: animals, cars, ships
Pre-trainedthe starting point
An 8 by 8 grid of almost blank white images
α = 0nearly blank
An 8 by 8 grid of simple, smooth images of animals, planes and objects
α = 0.5simpler images
An 8 by 8 grid of varied images close to the pre-trained ones
α = 10close to the start

Even digits, MNIST τ = 10

An 8 by 8 grid of sixes only
α = 0one digit
An 8 by 8 grid of mostly sixes with some zeros, twos, fours and eights
α = 0.3more variety
An 8 by 8 grid of varied even digits
α = 0.8still more

Paper, Figs. 3 and 4. Curves traced from Fig. 4e and 4f; dashed: the pre-trained model.

The trade-off is set with one number, not with a new dataset.

The evidence Finding 04

It aligns Stable Diffusion 3, and keeps it diverse.

Fine-tuned on “a cat in the sky” with a CLIP reward, it scores highest of all methods compared, with diversity close to the base model. The W2 bound lifts every method it is added to.

Prompt: “a cat in the sky”

Three different cats in the sky: one resting on a cloud, one leaping between clouds, one floating
ORW-CFM-W2, oursCLIP score 35.46  ·  diversity score 4.21

Paper, Table 1 and Fig. 8. Scores averaged across 64 samples; diversity is the mean pairwise distance between CLIP embeddings. Three samples per method, as shown in the paper.

Spatial and compositional prompts

A banana on the left of an apple

SD3

Two SD3 samples: bananas lying around and behind an apple

ORW-CFM-W2

Two samples after fine-tuning: a banana on the left, an apple on the right

A cat on the left of a dog.

SD3

Two SD3 samples: a single dog, and a single animal that mixes cat and dog

ORW-CFM-W2

Two samples after fine-tuning: a cat on the left sitting beside a dog on the right

A laptop on top of a teddy bear

SD3

Two SD3 samples: a laptop beside a teddy bear

ORW-CFM-W2

Two samples after fine-tuning: an open laptop resting on top of a teddy bear

A stack of 3 cubes. A red cube is on the top, sitting on a red cube. The red cube is in the middle, sitting on a green cube. The green cube is on the bottom.

SD3

Two SD3 samples: stacks of four cubes with the colours in the wrong order

ORW-CFM-W2

Two samples after fine-tuning: two red cubes stacked on a green cube

A stack of 3 plates. A blue plate is on the top, sitting on a blue plate. The blue plate is in the middle, sitting on a green plate. The green plate is on the bottom.

SD3

Two SD3 samples: piles of several blue and green plates

ORW-CFM-W2

Two samples after fine-tuning: blue plates stacked on a green plate

An emoji of a baby panda wearing a red hat, green gloves, red shirt, and green pants

SD3

Two SD3 panda emojis with a red hat, green gloves and a red and green body that runs together

ORW-CFM-W2

Two panda emojis after fine-tuning, each with a red hat, green gloves, a red shirt and green pants

Paper, Figs. 7 and 10; prompts from DrawBench as used in DPOK, with the paper's own wording. Fig. 7 samples every method with the same random seeds.

Better placement and attributes, without giving up image quality.

Why it matters Finding 05

Change the reward, not the dataset.

It needs only a score for each image it makes: no human‑collected data, no filtered datasets, no gradient of the reward.

Stable Diffusion 3, “a train on top of a surfboard”, with three other reward models

HPS-V2

Three samples with the HPS-V2 reward: a train on tracks, a locomotive by the sea, a locomotive on a surfboard in the wavesThree more samples with the HPS-V2 reward: locomotives riding surfboards on the water

Pick Score

Three samples with the Pick Score reward: trains on surfboards on a beach and in the surfThree more samples with the Pick Score reward: locomotives on surfboards in the sea

Alpha Clip

Three samples with the Alpha Clip reward: a surfboard on the sand, a train by the water, a train on a surfboard on the beachThree more samples with the Alpha Clip reward: trains riding surfboards on the water

Paper, Fig. 9, in the paper’s order.

What this opens up

Any score will do

A classifier, a JPEG file size, CLIP, HPS‑V2, Pick Score, Alpha Clip: one method, no reward‑specific changes.

Its own data

Every training image is one it generated itself. Flip the sign of the reward and the same model learns odd digits instead of even ones.

Two dials, clear roles

τ sets how fast it converges and α where it settles, so fine-tuning stays under control.

The film

The story, as a short film.

Opens the film page.

Cite

Cite the paper.

International Conference on Learning Representations 2025 (ICLR 2025).

@inproceedings{fan2025online,
  title={Online Reward-Weighted Fine-Tuning of Flow Matching
  with Wasserstein Regularization},
  author={Jiajun Fan and Shuaike Shen and Chaoran Cheng
  and Yuxin Chen and Chumeng Liang and Ge Liu},
  booktitle={The Thirteenth International Conference on
  Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=2IoFFexvuw}
}