TL;DR — Reinforcement learning turns one base model into specialists, each strong in one domain and behind in the others. Putting them back into one model has meant training again: on-policy distillation costs rollouts, teacher passes and gradient steps for every teacher. We show that this step does not need training. When the specialists share a base model, distillation has an exact layer-wise solution: one forward pass over a few hundred of each specialist’s own answers, then a least-squares solve per layer. The fused model is, on average, more accurate than either specialist.

WATCH THE FILM

The problem: one model again means training again

A family of specialists that share a base model is now the usual result of reinforcement learning: one is strong at mathematics, another at code, and each falls far behind in the other’s domain. Getting one model that does both has meant another round of training, through multi-domain RL or on-policy distillation.

On-policy distillation

The student generates, every teacher scores the text, and gradients flow: rollouts, teacher passes and a backward pass at every update, for every teacher.

TF-OPD

Ask each layer of the fused model to reproduce each teacher on the inputs that teacher sees while reading its own answers. That is least squares, solved exactly. Forward passes and linear algebra only.

Method: four steps, no gradient

Each linear layer of the fused model must reproduce every specialist’s output on the inputs that specialist receives while it reads its own answers. That is a least-squares problem, and its solution needs only the second moments Gd of those inputs:

W★ = Wb + (Σd Δd Gd)(Σd Gd)−1

Here Δd is specialist d’s update to the base weight Wb (the paper also shrinks the off-diagonal second moments slightly). Task arithmetic applies every specialist’s update to every input, including inputs only another specialist ever sees; TF-OPD minimizes exactly that error, weighted by how often each input direction occurs.

TF-OPD: four steps, no gradient Four stages. Each specialist answers a few hundred training problems; answers stuck in repetition loops are dropped; each specialist reads its kept answers once while every linear layer records the second moments of its inputs; each layer is then solved in closed form. The result is one model of the same size, with no gradient step. 1 Rollouts their own answers 2 Selection drop looping answers 3 Statistics one forward pass 4 Solve each layer, exactly math code 256 problems each dropped keep: correct math, tested code G = E[x xᵀ] W★ = Wb + (Σd ΔdGd)(Σd Gd)−1 0 gradient steps ✓ one model, same size forward passes and linear algebra only: no backpropagation, no optimizer, no learning rate
How to read it. Each specialist answers a few hundred of its own training problems; answers stuck in repetition loops are dropped (for mathematics, only correct answers are kept, for code only answers that pass their tests); each specialist reads its kept answers once while every linear layer records the second moments of its inputs; then each layer is solved exactly. Schematic.

The statistics are on-policy: they come from each specialist reading its own answers, which are the inputs a layer actually sees in use, not from reference solutions the model never wrote. The procedure keeps no optimizer state and has no learning rate to tune; for two specialists it takes about 10 GPU-hours of forward computation in total, on 48 GB GPUs.

Results

On Olmo 3 7B, with a mathematics specialist and a code specialist trained by RL from the same base, the fused model has the highest mean accuracy of the four models: built without a gradient step, it is on average better than either specialist it was built from.

61.1
mean accuracy over seven math and code benchmarks, highest of all four models
50.0
AIME24, level with the math specialist
69.4
OlympiadBench, highest of all four models
0
gradient steps; about 10 GPU-hours of forward passes in total
Mean accuracy over seven benchmarks Mean accuracy over seven mathematics and code benchmarks on Olmo 3 7B: TF-OPD 61.1, math specialist 59.8, code specialist 53.6, base model 31.6 percent. TF-OPD (ours) no training 61.1 Math specialist RL on math 59.8 Code specialist RL on code 53.6 Base model Olmo 3 7B 31.6 010203040506070 mean accuracy over seven benchmarks (%) Mean accuracy over seven benchmarks Mean accuracy over seven mathematics and code benchmarks on Olmo 3 7B: TF-OPD 61.1, math specialist 59.8, code specialist 53.6, base model 31.6 percent. TF-OPD (ours) · no training 61.1 Math specialist · RL on math 59.8 Code specialist · RL on code 53.6 Base model · Olmo 3 7B 31.6 010203040506070 mean accuracy over seven benchmarks (%)
Figure 1. Mean accuracy over MATH-500, OlympiadBench, AIME24, AIME25, HMMT25, HumanEval+ and MBPP+ (Olmo 3 7B).

Each specialist is far ahead in its own domain and far behind in the other. TF-OPD stays close to the specialist of the domain on every benchmark and keeps most of what reinforcement learning added on each, so it is the only model that is never far from the best score.

Table 1. Accuracy (%) on Olmo 3 7B. AIME24, AIME25 and HMMT25 average four sampled answers per problem; the other benchmarks use one. Released checkpoints are evaluated in bfloat16 and the fused model in float16. Best value in each column in bold.
ModelMATH-500Olympiad­BenchAIME24AIME25HMMT25HumanEval+MBPP+Mean
Base model63.433.415.810.08.345.744.231.6
Math specialist91.868.750.043.330.871.362.459.8
Code specialist85.662.526.730.015.086.069.353.6
TF-OPD (ours)90.869.450.038.327.584.167.561.1

The released specialists are scored in bfloat16, their native format. Scored in the same format (float16), the fused model is within a point of the mathematics specialist on AIME25.

The surprise: the data matters more than the solver

Most of a specialist’s answers are ordinary reasoning that ends with a result. A few never end: the model starts repeating itself and continues to the length limit. For the mathematics specialist these are a small share of the answers but, being the longest, a large share of the tokens, and every token counts equally in the statistics.

A few looping answers carry many of the tokens Of the mathematics specialist's rollouts, answers that end in a repetition loop are 7 percent of the answers but 23 percent of the tokens. of the answers 7% loop ordinary reasoning of the tokens 23% loop ordinary reasoning A few looping answers carry many of the tokens Of the mathematics specialist's rollouts, answers that end in a repetition loop are 7 percent of the answers but 23 percent of the tokens. of the answers 7% loop ordinary reasoning of the tokens 23% loop ordinary reasoning
Figure 2. The mathematics specialist’s rollouts: answers that end in a repetition loop, as a share of the answers and of the tokens.

Remove them, and the fused model goes from task arithmetic’s level on AIME25 to the specialist’s own level in the same format. The damage does not show token by token: built from all answers, the fused model already predicts almost the same next token as the specialist, yet a long derivation is decided at a few of its steps.

AIME25 by the answers the statistics use AIME25 accuracy in float16. TF-OPD with looping answers removed scores 40.0 percent, TF-OPD with all answers 34.2, task arithmetic 34.2; the mathematics specialist in the same format scores 39.2. TF-OPD, loops removed ours 40.0 TF-OPD, all answers same solver 34.2 Task arithmetic weights only 34.2 math specialist, float16: 39.2 01020304050 AIME25 accuracy, float16 (%) AIME25 by the answers the statistics use AIME25 accuracy in float16. TF-OPD with looping answers removed scores 40.0 percent, TF-OPD with all answers 34.2, task arithmetic 34.2; the mathematics specialist in the same format scores 39.2. TF-OPD, loops removed · ours 40.0 TF-OPD, all answers · same solver 34.2 Task arithmetic · weights only 34.2 01020304050 AIME25 accuracy, float16 (%) dashed: math specialist, float16: 39.2
Figure 3. AIME25 accuracy (%), four sampled answers per problem, every model in float16. Only the answers that enter the statistics change between the two TF-OPD rows; the final model of Table 1 also keeps only code answers that pass their tests.

Case study: where the specialist loops

The fused model is not only a compromise between its sources: on some problems it succeeds where every sample of that domain’s specialist fails.

HMMT25: In a two-dimensional cave with a parallel floor and ceiling, two stalactites of lengths 16 and 36 hang perpendicularly from the ceiling, while two stalagmites of heights 25 and 49 grow perpendicularly from the ground. If the tips of these four structures form the vertices of a square in some order, compute the height of the cave. (Answer: 63)

TF-OPD · 3 of 4 samples correct

“Suppose that the four points form a square. Therefore, the midpoints of the diagonals must coincide.” … “their y-coordinates must satisfy H − 26 = 37 ⇒ H = 63.”

63 ✓

Math specialist · 0 of 4 samples correct

“… which implies 400 = 576, which is impossible.” … “I think I have to concede and say the answer is 61.” (the same opening then returns in later steps)

61 ✗

The specialist’s failure has the shape of the looping answers found among its training rollouts; the fused model, built without those answers, does not follow it there.

When to use it

For specialists that share a base model, the consolidation step after reinforcement learning may not need training. Adding a specialist costs its rollouts and one more term in each layer’s sums; nothing is retrained. When the teachers differ from the student in architecture or size, training-based distillation remains the tool.

BibTeX

@article{yang2026tfopd,
  title   = {Training-Free On-Policy Distillation},
  author  = {Yang, Bangji and Fan, Jiajun and Zhu, Zhaoyu and
             Ma, Hongbo and Liu, Ge},
  journal = {arXiv preprint arXiv:2610.22236},
  year    = {2026}
}