Training-Free On-Policy Distillation
Published in arXiv preprint, 2026
Recommended citation: Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu. "Training-Free On-Policy Distillation." arXiv preprint arXiv:2610.22236, 2026. https://arxiv.org/abs/2610.22236
Specialists fine-tuned from one base model can be fused into a single model with forward passes only: each linear layer of the fused model has an exact least-squares solution that needs only second moments of its inputs, gathered from a few hundred rollouts. The fused model is, on average, more accurate than either specialist alone.