"A model learns better from data it generates itself (on-policy)." This has become a go-to piece of conventional wisdom lately. The trouble is that most of the evidence for it comes from comparisons that pit two wildly different training recipes against each other all at once. When a Cambridge team pulled the variables apart one by one and looked again, the thing that decided the outcome turned out not to be "on-policy or not." Final performance was governed by how a student checks its answers against the teacher (the KL direction), and how much the model forgets its prior skills and how much it gets rewritten were governed by the learning rate. Much of what we'd been crediting to on-policy, it turns out, was really the work of other settings.
Let me define terms first. LLM post-training is the stage where a pre-trained model gets tuned for real-world use. Here, the approach in which a model learns from its own outputs is on-policy (the canonical example being RLVR — reinforcement learning that rewards a model based on whether its answer is correct), while the approach that learns from a fixed, pre-built dataset is off-policy (the canonical example being SFT — supervised fine-tuning, where the model is taught to copy correct examples directly). The conventional wisdom holds that on-policy is better on several fronts: it forgets less of what it learned before, changes the model's internals less, and copes better with problems it was never trained on.
But the evidence for this is mostly comparisons that pit SFT against RLVR wholesale. The two methods don't differ only in "who generates the data." They differ in the training objective, the reward signal, how much of the answer is revealed, the scale of optimization, and the learning rate — everything. In cooking terms, it's like swapping the ingredients, the heat, the cook time, and the seasoning all at once and then declaring, "yep, it's the pan that makes the difference." It becomes genuinely hard to isolate the contribution of the pan alone — that is, of who generates the data.
Why does this matter in practice? On-policy is expensive, because the model has to keep generating fresh answers throughout training. Off-policy just reuses data you've already built. If on-policy's advantages actually come from other design choices, the case for paying that steep cost gets a lot weaker.
So how do you isolate the contribution of the pan alone? The team's clever move was to set aside the messy SFT-vs-RLVR comparison and use distillation — transferring knowledge from a strong teacher model to a weaker student model — as the test bench. In this setup, you can cleanly change who generates the data while leaving the rest of the training process untouched.
There are three variables to isolate:
① Rollout policy — whether the training data is generated by the teacher (off-policy) or the student (on-policy). ② KL direction — how the student's answers are checked against the teacher's. ③ Learning rate — how hard the model is corrected in a single step.
Where prior work changed all three at once and looked at the result, this team moves just one at a time to pin down the real cause.
The most confusing of the three is the KL direction. Let's build the intuition first. Picture two students. One says, "I won't miss a single thing the teacher taught," and follows every point the teacher emphasized, evenly. The other says, "I'll nail the one thing I'm most confident about," digs deep in just one spot, and ignores whatever the teacher glossed over, just as the teacher did. The former has the character of forward KL; the latter, reverse KL.
Now let's attach the terms. KL is a measure of how different two probability distributions are; here it captures the gap between the student's and the teacher's beliefs about "which word (token) to produce next." Forward KL heavily penalizes the student for missing tokens the teacher assigned high probability to (it tries to cover all of the teacher's "peaks" — hence mode-covering). Reverse KL does the opposite: it penalizes the student for putting probability on tokens the teacher considered unlikely (it concentrates on a single peak — hence mode-seeking).
Until now, people have generally lumped on-policy together with reverse KL, and off-policy together with forward KL. But "who generates the data" and "how you check the answers" are separate choices. Because prior work paired them up in its comparisons, it couldn't isolate the real cause. So the team deliberately breaks the pairing and runs all four combinations (on/off × forward/reverse). On top of that, they fix the learning rate separately, erasing one more hidden confounder.
Here's the experiment itself. The main runs distill a Llama-3.1-8B teacher into a Llama-3.2-1B student (the appendix reproduces this with Qwen2.5-7B → Qwen2.5-1.5B). The teacher and student come from the same model family because only when the tokenizer and vocabulary match can you line up their distributions directly at the token level. The catch is that same-family models can share so much capability that there's little left to learn — so for each task the team trained a dedicated teacher to widen the skill gap, then froze it throughout distillation. The tasks span three domains: medical reasoning (MedReason), scientific reasoning (Science), and arithmetic reasoning (Countdown). The student was full-parameter fine-tuned for 150 steps, and each configuration was repeated across 3 random seeds.
There are three metrics:
What this experiment can and can't isolate
This test bench cleanly measures "what happens when you change only who generates the data." Conversely, it does not fully explain the performance gap that real-world SFT and RLVR exhibit. In practice the two methods also differ along other axes, like the reward signal and how much of the answer is revealed. This paper isolates only the portion of that gap attributable to the rollout policy. The conclusions below should be taken strictly within that scope.
Final accuracy — there was no on-policy advantage. Averaged across the three datasets, peak accuracy was 72% for on-policy distillation (OnPD) and 73% for off-policy distillation (OffPD) — essentially the same (Countdown uses the Countdown-3 variant). The KL direction, by contrast, split things sharply. The diligent student (forward KL) held steady at 71–73% regardless of where the data came from or what the learning rate was. The one-well-digging student (reverse KL), meanwhile, swung from 35% all the way to 72%, and was especially sensitive to the learning rate. What decided performance was not "who generates the data" but "how you check the answers."
Forgetting — the deciding factor was the learning rate. Similar accuracy can still hide different amounts of "forgetting" underneath. On inspection, OOD performance dropped by at most 1.3 percentage points when the learning rate was low, but by a full 11.2–14.0 points when it was high. The difference due to rollout policy was negligible by comparison. Strikingly, with forward KL, lowering the learning rate made most of the forgetting disappear while in-distribution (ID) performance held — the trade-off where "you must accept forgetting to gain performance" was not inevitable. What governed forgetting was not whether training was on-policy, but the learning rate.
Sparsity — the learning rate again. The same picture repeats. Update sparsity was 85.3–89.6% at low learning rates and 51.9–60.0% at high ones. The gap between rollout policies was far smaller; if anything, in every pairwise comparison off-policy touched the model just as little as on-policy did. The conventional wisdom that "only on-policy perturbs the model less" did not hold here.
So when does the rollout policy matter? Let's call the two students back. The diligent student (forward KL) keeps learning to close even a small gap from the teacher, so it trains stably no matter which route the data arrives by. The one-well-digging student (reverse KL), however, simply doesn't look at what it doesn't already know — and at those points the learning signal vanishes. That's why it only comes into its own when it learns from paths it worked out itself, i.e., on-policy data.
The results matched this prediction. When the team varied the data source continuously from the teacher side to the student side, forward KL held above 80% holdout accuracy across the entire range, with a swing of just 5.2 percentage points — it even learned while passing through the jumbled middle region. Reverse KL behaved differently: at low learning rates it did well on student-side (on-policy) data and poorly on the teacher side, and at high learning rates it was unstable or occasionally collapsed.
Where on-policy genuinely shone. None of this means on-policy is useless. It has an advantage that pass@1 (one draw, one shot at correctness) simply can't capture. With forward KL, the gain from drawing multiple samples (pass@k, i.e., output diversity) was larger on ID tasks. And on generalization to harder Countdown variants, on-policy data helped consistently under both KL directions. That said, the authors add an honest caveat: this early generalization advantage did not reliably survive a subsequent round of RLVR. And when paired with reverse KL, on-policy data reduced the degree to which the teacher's speaking style rubbed off on the student.
Start with the limits the authors acknowledge themselves. The point that on-policy's generalization advantage doesn't stably survive a later round of RLVR is stated plainly in the paper. They also shored up their conclusions along three lines of defense — rechecking them with gradient clipping (the safety mechanism that caps gradients so they don't blow up) removed, with KL estimated from samples, and on tasks requiring longer reasoning.
In my view, the biggest limitation is one of scope. This clean test bench is built for "strong→weak distillation" where teacher and student share a vocabulary. Whether the conclusions carry over to distillation across different families or vocabularies — or to pure RLVR that isn't distillation at all — is outside what this paper claims. The models are also centered on 1B–1.5B-class students, so whether the same asymmetry holds for much larger students is another open question. The result that "on-policy helps generalization to harder variants" came mainly from the Countdown arithmetic task, so it's safest not to extrapolate it to other domains.
Distilling a big model's skills into a small, cheap model to run as a dedicated worker is a staple task at ONTLE. Say we hand a small model the automated inquiry triage for a small-business client — sorting incoming inquiries by type and drafting a first-pass reply in a set tone. The conventional wisdom would say run it on-policy, with the student generating fresh answers at every step — and the generation cost rides along for the entire training run. This paper makes us reconsider that choice by splitting it into two situations.
Situation A — when the worker only needs to handle familiar, repetitive work (jobs with a steady shape, like inquiry classification and first-pass replies)
Situation B — when we need to push into harder problems not seen during training
So the first question a PM should ask isn't "on-policy or not," but "does this worker only need to handle familiar work, or must it also cope with unfamiliar difficulty?" When weighing cost against quality at ONTLE, the default is Situation A, and on-policy is the card we play for Situation B. For the inquiry-triage project right now, run it once more with the Situation A configuration and decide by laying its holdout accuracy and forgetting margin side by side against the on-policy version.