This paper digs into why a small language model, while learning to imitate a larger teacher through On-Policy Distillation (OPD), ends up producing abnormally long responses that burn through the entire generation budget (the length limit). The authors pin down one of the core causes: the student and teacher place their probability mass on different termination tokens (EOS) when ending a response—in other words, a termination-token mismatch. They further show that merely aligning the declared set of stopping tokens is not enough; once you treat functionally equivalent EOS tokens as "one semantic stopping action," the length blowup drops sharply across all three model families: Qwen3, Llama, and Gemma.
The go-to method for making language models smaller and faster is distillation: you train a small model (the student) to mimic the decisions of a big model (the teacher). Among the variants, On-Policy Distillation (OPD) has the student generate text itself, then learn by comparing, in real time, what the teacher would have done during that generation. The student walks on its own two feet while the teacher corrects it from the side.
But something odd happens here. The student's answers keep getting longer. It keeps writing past the point where a short answer would do, sometimes exhausting the preset maximum length (the generation budget). This is the "length inflation" the paper is about.
Here is an analogy. When wrapping up a whole message, some people sign off with "that's all from me," while others say "I'm done here." Both do the same thing—signal that the response is over—but the outward wording differs. Suppose the teacher has a habit of closing with "that's all from me," while the student originally liked to close with "I'm done here." OPD tells the student, "suppress that closing you favor," yet it fails to reliably hand over the teacher's preferred closing. As a result, the student loses its sense of how to wrap up and just keeps going.
Previously, this kind of problem was handled loosely—with a length penalty or some hyperparameter tuning. What sets this paper apart is that it traces the root cause to a concrete mechanism: a mismatch in how responses are ended (the termination token).
Here is the key fact the authors uncovered. The student (a pretrained base model) and the teacher (a post-trained model that has gone through additional fine-tuning) can place their probability mass on different EOS tokens in practice—even when their declared stopping sets (the special tokens designated for ending the response) are completely identical. This phenomenon shows up across all three model families: Qwen3, Llama, and Gemma.
Why is this mismatch a problem? OPD suppresses the termination behavior the student originally preferred, but the teacher's preferred alternative termination token is never stably transferred to the student. In other words, the act of "stopping" itself slips between the two models, so the student never gets a clear signal about when to end its response, and its answers drag on.
The question the authors raise here matters: "Then can't we just align the decoding stopping set between the two models?" The experiments say that alone is insufficient. Matching the lists does not mean the two models actually load probability onto the same token.
So the method they propose is to treat functionally equivalent EOS tokens as a single, shared semantic stopping action. Going back to the analogy: whichever sign-off is used, both are bundled together as the same-meaning act of "ending the response," so that regardless of how the student closes, the intent to "stop" is properly conveyed between teacher and student. They align on function (what it does) rather than surface form (which token it is).
This approach substantially mitigated the mismatch-driven length blowup across all three model families.
The abstract reports three main findings (since no concrete numbers are given in the abstract, I summarize them qualitatively).
| Attempt | Result |
|---|---|
| Aligning only the stopping-token set | Insufficient—the mismatch persists |
| Treating functionally equivalent EOS as a shared stopping action | Length blowup substantially mitigated across Qwen3, Llama, and Gemma |
| Stage-by-stage OPD analysis with K2-Horizon | Termination preference shifts significantly during training; a separate length blowup remains in the later phase |
The authors also analyzed OPD across multiple training stages using something called K2-Horizon, to see how termination behavior changes over the course of training. This stage-by-stage analysis revealed two things. First, termination preference can shift significantly as training progresses—a model's habit for "when to stop" is not fixed but keeps changing during training. Second, in the later phase of OPD there is another length blowup that persists beyond termination alignment.
This second finding is what makes the paper honest. The authors state plainly that termination mismatch is an important, but not exhaustive, cause of the length problem. They acknowledge that there is clearly a residual portion their identified cause does not explain. They also note that they have released an implementation incorporating the proposed termination-handling correction.
ONTLE is a SaaS that dispatches AI project managers, and we run multiple language models inside real product pipelines. This paper puts its finger precisely on a practical trap you commonly hit when doing distillation work to "shrink a big model and cut costs."
First, response-length blowup means cost and latency. When an AI PM summarizes meeting notes or classifies tasks, needlessly long answers raise token costs and slow down responses—and in the worst case the answer gets cut off at the length limit. This paper tells us the cause may lie in an easy-to-miss layer: not the quality of the prompt, but a mismatch in the model's signal for ending its response.
Second, "aligned on the surface" is not the same as "actually aligned." The finding that the real probability distributions can diverge even when the stopping-token lists are declared identically is a caution, from a product standpoint, against the complacency of "we matched the config values, so we're done." After fine-tuning or distillation, you have to verify by observing the actual output behavior (whether answers end properly), not the declared settings.
Third, behavior shifts at every training stage. The observation that termination preference changes significantly during training suggests you cannot train a model once and be done—you have to monitor output characteristics stage by stage. In an environment like ONTLE that rolls out multiple models, "check the response-length distribution after distillation/fine-tuning" is worth adding to the pre-deployment checklist.
In short, rather than flashy performance gains, this work honestly digs into the root of a practical headache—"why does the small model keep failing to stop talking?" For any team that puts models directly into a product, it offers a practical lesson: keep a close eye on the termination behavior that drives cost and quality.
Ordinary distillation is closer to having the student learn by imitating answers the teacher model prepared in advance. On-Policy Distillation, by contrast, has the student generate text itself and learn by comparing, in real time, how the teacher would have judged during that generation. Because the student is corrected on the very sentences it produced, the student's own habits (such as how it ends its responses) have a big influence on learning.
OPD suppresses the termination style the student originally preferred. But if the teacher's preferred alternative termination token is not stably transferred to the student, the student loses its signal for "how to stop." The stopping behavior gets suppressed while no replacement behavior comes in, so the student keeps extending the response and eventually uses up the length limit.
According to the abstract, that alone is insufficient. Even if the two models' declared stopping-token sets are completely identical, the token they actually load probability onto can differ. So instead of matching the lists, the authors proposed treating functionally equivalent EOS tokens as "one semantic stopping action," and this proved effective across the three model families.
No. The authors clearly state that termination mismatch is an important cause but not the whole story. In the later phase of OPD training, another length blowup remains that persists even after termination alignment, and its cause is an open problem this paper does not fully explain.