Say you're building a service where an AI generates complex financial-product quotes and digs up the right policy clauses for a customer's question. You prototyped it with the smartest, largest model available. But once you're running tens of thousands of these a day, the operating cost becomes untenable.
The go-to move here is distillation. It's the technique of copying the judgment of an expensive large model (the teacher) into a cheap-to-run small model (the student). It's one of the biggest cost-cutters in any AI project.
Things get tricky when the two models come from different families. When even the way they chop up text — the tokenizer (the component that splits a sentence into token units) — differs, transferring knowledge gets hard. So engineers have long followed one piece of conventional wisdom: the more points you can line up for comparison, the better the training goes.
Researchers at Alibaba flipped that wisdom on its head. When they added extra supervision that filled the comparison range to completion, accuracy dropped in every single one of the 18 configurations they tested. Being able to compare a lot and whether that comparison actually helps training turned out to be two entirely different things.
Even when two models receive the same sentence, they cut it at different places. One model sees "Hello there" as a single chunk; another splits it into "Hello" and "there." If the cut points match, you can compare word against word directly. If they don't, you can only compare by bundling up a "whole span." It's for the same reason that, even on an identical sentence, the probabilities for the next token sit on top of two different vocabularies.
So why does this particular paper focus on an approach called "OPD"? Because it's the most talked-about trend in distillation right now. OPD (On-Policy Distillation) doesn't have the student copy out answers the teacher prepared in advance. Instead, it lets the student generate sentences on its own and then corrects it along that path. Every time the student wanders off, the teacher is right there tutoring it one-on-one — "this way." Because you teach by following the path the student actually walks, it works well, and distillation research has been converging on it lately. This paper drills straight into that OPD setting (the paper also interprets OPD as a form of reinforcement learning).
Cross-tokenizer OPD therefore needs two pieces of alignment work. One is matching up where the sentence gets cut; the other is deciding how to compare two probability distributions over different vocabularies. Research so far has flowed in one direction: find some way to recover more of the comparison points that got thrown away because the cut points didn't match. The methods varied, but the premise was the same. More is better.
The paper starts by splitting token alignment into two kinds. When one student token and one teacher token cover exactly the same span, that's a strict group. When one side has to spend several tokens to produce the same span, that's a mismatched group. What prior work struggled to recover is exactly this mismatched group — the part that was being discarded.
But the discarded part turned out to matter less than everyone thought.
The two models' vocabularies overlap by only 39.49–64.87%. On paper, that's roughly half. But look at the sentences the student actually generates, and the story changes. In those sentences, the strict group covered 85.57–96.98%. In other words, even though the cut points differ from vocabulary to vocabulary, in the sentences the student actually produces, most of the positions line up cleanly with the teacher. On-paper overlap and real-world operational alignment were completely different numbers.
The same held for the probability distributions. At strictly aligned positions, the vocabulary the two models share held almost all of the teacher's probability (in the 99% range) and the student's probability (also in the 99% range). Narrow it further. Looking at just the top 16 shared tokens ranked by the student's probability, that handful already contained more than 93.54% of the teacher's probability and 94.55% of the student's. Expanding from 16 up to 128 added at most 3.44 percentage points on the teacher side and 2.70 on the student side. The returns dropped off sharply the wider you went.
One detail stands out here. These top 16 were chosen by looking at the student distribution alone, and yet most of the teacher's thinking was already inside them. That is: just the 16 tokens the student flagged as "these are the likely ones" nearly coincide with where the teacher is looking. So the mechanism that keeps the student's predictions from straying far from the teacher's intent (reverse KL) was also fine running over just those 16. It held up well enough to rival using the entire shared vocabulary — though what the paper showed is that narrowing this far retained over 96% of the performance gain, not a proof that the training signal itself is equivalent.
Now to the experiments. The researchers paired teachers and students across different families in three combinations: Qwen→Llama, Granite→Phi, and Granite→Qwen. All three cross company lines, so their tokenizers are entirely misaligned. On the student side they included both a raw base student that hadn't been touched and a student that had already been refined (instruction-tuned). The design is meant to show the conclusion isn't specific to one kind of student. For the tasks, they chose math and code reasoning, where the right answer is sharp. Clean grading is what lets you precisely measure the alignment structure of the path the student walks.
Before reading the table, let's be clear on what the numbers mean. The values below are the average score (%) you get by having the student actually solve the tasks. Higher is better. The top row, "before distillation," is the small model's starting score before any distillation.
| Condition | Qwen→Llama | Granite→Phi | Granite→Qwen |
|---|---|---|---|
| Before distillation | 26.96 | 36.55 | 29.56 |
| Full shared vocabulary | 32.86 | 42.49 | 47.35 |
| Top 16 | 32.64 | 42.26 | 47.06 |
That distillation lifted all three pairs above their starting points is no surprise. Two things are worth noting.
First, compression barely cost any performance. Distilling with just the top 16 captured over 96% of the performance gain (relative to pre-distillation) that you got from using the full vocabulary. Going from 16 to 128 produced no consistent extra benefit. The compression method the researchers put forward beat all four competing methods that came before it. Among those, the top-16 setting scored 0.51–1.05 percentage points higher on average than even the strongest competitor.
Second, widening the range actually made things worse. Leaving the strict-group supervision in place, the researchers piled on one more mechanism to recover the mismatched group — one that makes the overall span probabilities match each other even on spans where the cut points differ (a span MSE loss). This generates a signal for every previously discarded span, so the comparison range is structurally filled to 100%. By conventional wisdom, that should help.
They ramped the weight on that mechanism up from 0.25 (0.25, 0.5, 0.75, 1.0, 1.25, 1.5), and all three pairs peaked when the weight was 0 — that is, with no extra supervision added at all. All 18 configurations with a positive weight came in 0.27–1.20 percentage points below their own baseline, without exception. They filled the range to 100%, and the scores went down.
The researchers looked at the gradient — the direction training pushes the model in. They examined both which way the signal points (direction) and how hard it pushes (magnitude).
It's easier to picture two advisers. One (the strict-group supervision) steadily points north. That's the right direction. The other (the supervision they added) keeps pointing east or west. And as training progresses, this second adviser's voice keeps getting louder.
Looking at the actual training process, the two advisers really were at odds. Measured at a point trained on the strict group alone, the direction the extra supervision pointed was nearly perpendicular to the strict supervision (the cosine between the two directions sitting near 0) or even pointed the opposite way. And the magnitude ratio of the two signals grew as training went on. The misaligned adviser pushed harder and harder.
They kept a control, too. Even a trustworthy signal pointing north, when split in two and measured half against half, won't point in perfectly identical directions. It's the natural divergence you get from splitting a single purpose. Yet the divergence of the extra supervision exceeded even this natural divergence at every point. A signal whose direction is off while its force only grows. The very signal meant to let you compare more turned out to be, for training, noise — or a force pulling the other way.
Start with the lines the researchers draw themselves. The gradient diagnosis is framed cautiously, as something that "may help explain." They showed only that the phenomenon of a diverging direction and a growing force co-occurs with the accuracy drop — not that it's the cause. The implementation of the extra supervision is also just one form: an MSE over span probabilities. The researchers' own logic leaves room for a different result if the mismatched group were handled some other way.
It's also worth being clear that the tasks skew toward math and code. These are domains where answers can be graded, so the signal is sharp. Whether compressed supervision is just as sufficient on tasks with fuzzier answers — tone, style, long dialogue — is outside this experiment. The appendix reports that even when the scope is widened to a 235B teacher distilled into an 8B student, and to ALFWorld (an agentic task involving carrying out actions), the top-16 approach was still the highest; but extending it to open-ended generation in general needs more evidence. The "16" in top-16 is itself an empirical value from these three pairs on these tasks. For combinations with very different vocabulary overlap or probability concentration, you'd have to measure it again for yourself.
Let's narrow to a single scene — that financial-quote service from earlier. You validated the step-by-step reasoning logic of the quote generation on a large model, and now you want to move it onto a small model with low operating cost. But the two models are from different families, so their tokenizers differ.
Follow the conventional wisdom and your design doc says: "the tokenizers differ, so let's lay down a heavy pipeline that aligns and recovers as many tokens as possible." We'd shake that premise first. The first thing to measure isn't what percentage of the vocabularies overlap. It's how well the cut points line up, and how concentrated the probability is on the shared vocabulary, along the actual generation path of this quote-reasoning task. If that concentration comes out high — as it does for reasoning tasks — then instead of an expensive full-recovery pipeline, you put "compressed distillation using only a small number of top candidates at strict positions" on the table as your first draft.
Let's draw one line clearly. What the paper showed with data goes as far as "on reasoning tasks, extra supervision that widens the recovery range can cut accuracy." Turning that into "operating cost for quotes drops by X" is practical inference, not a measured result. Still, the direction is clear. Before laying down a heavy pipeline as the default, slotting in a single checkpoint — "measure the cut points and probability concentration on a small sample before the work" — is cheaper than laying it down broadly and rolling it back later. If the measurement doesn't come out as sharp as it does for reasoning, that's when you consider wider supervision.
So the lesson this research leaves for the field is to move where the counterweight sits. The counterweight should now shift from "how much can we compare (coverage)" to "is that comparison trustworthy for training (reliability)." Deferring the judgment until after the measurement — that's the heart of it.
Because at every step during training, the thing immediately at hand is the student's prediction. The premise is the observation that most of the teacher's probability is already inside the student's top 16. So you don't have to go to the trouble of selecting a separate set of candidates based on the teacher side, and the computation stays that much simpler.
What this paper showed goes only as far as "matching span probabilities with an MSE" being harmful. It isn't a proof that the mismatched group itself is useless. The researchers leave room for a different result with a different method. That said, on reasoning tasks, most of the path the student actually walks is already strictly aligned, so a trustworthy signal was already sufficient right there. That's how weak the reason to go out of your way to recover the mismatched group became. (Note, though, that this doesn't mean the probability mass of the mismatched spans is small — the probability concentration the paper measured is purely the concentration on shared vocabulary at strictly aligned positions.)
The two start from different states. Only if the same tendency appears in both an untouched model and an already-refined one can you say it's "a property of the alignment structure itself, not a quirk of the student type." Of the three pairs, Granite→Qwen is the base-student combination, and the same pattern showed up there too.