A language model's math and coding ability is shaped in large part by reinforcement learning. The trouble is that this training is expensive. Could the ability one model paid to acquire be passed to another model without retraining it from scratch? The answer from Alibaba Cloud's Waypoint-OPD is: don't copy wholesale. Read the intermediate training record, and transfer the final shift weighted by how consistently the two training segments pointed the same way.
Let's see what "transfer" means through a single scene. A teacher model is writing text, and at some position it hesitates between 'A' and 'B' for the next token. A token is the smallest unit a model works with when producing text — roughly a word fragment. Before reinforcement learning, it viewed A and B about equally. Reinforcement learning is iterative training where the model produces an answer several times, a grader tells right from wrong, and the model nudges itself toward whatever did well. Over thousands of rounds of this feedback, the teacher came to prefer A more and more. It is exactly this movement in preference — "they were about even, then it tilted toward A" — that gets handed to the student. Moving not the teacher's entire ability but only the part that training changed is what policy-shift distillation is. Because it transfers the trajectory of change rather than copying finished answers, the teacher doesn't even need to be smarter than the student.
The simplest approach compares just the two endpoints, before and after training. The authors call this Direct-OPD. The catch is that even when the endpoints are identical, the paths that got there can differ.
Look at A again. In one case the teacher's preference for A rose through training as 'low → medium → high' — a change that hardened steadily in one direction from start to finish. In another case it went 'low → even lower → high' — the teacher pushed A further away early on, then reversed course late and pulled it up. Seen from the end, both go from low to high, equally a case of "coming to like A more." But in the second, the late change undid the early change, which makes it that much less consistent a shift. Direct-OPD, looking only at the endpoints, cannot tell these two apart.
So Waypoint-OPD plants one more point in the middle. A single intermediate record can't reveal the entire path the teacher traveled. Still, it's enough to tell whether the early stretch and the late stretch are undoing each other or not.
The name of that middle point is the waypoint. The key is that the three moments are not three different models but three instants of the same teacher. They're saved three times within the same reinforcement-learning run: before training (Initial, step 0), midway (Waypoint, step 600), and at the end of training (Final, step 1200). All three are frozen and are not trained further during transfer.
The sequence goes like this. When the student, building a sentence, reaches some position, it narrows down a handful of next-token candidates it deems plausible (the top 16, in the implementation). The three moments of the same teacher each score these candidates. Each candidate then gets three scores, revealing two consecutive shifts: the early shift from Initial to Waypoint, and the late shift from Waypoint to Final. Joined together, they form the overall endpoint shift from Initial to Final.
Here, the direction judgment is made not candidate by candidate but by treating the entire shift made up of the top 16 candidates at one position (one generation site) as a single block. The early shift and the late shift at that position are each viewed as a single vector spanning the candidates, and how much the two vectors' directions overlap is measured by a cosine similarity weighted by the student's probabilities. What we did in the previous section — asking "did the preference for A harden in one direction, or bend midway and come back?" — is now measured once for the entire shift at that position. The more the two stretches overlap in the same direction, the more the endpoint shift at that position is treated as a trustworthy signal and given more weight. The weaker the overlap, the less weight; and when the direction has reversed — if the cosine similarity is at or below 0, except for a low-norm case where the change is too small to trust its direction — it is switched off entirely. The single value that results is applied equally to every candidate at that position. In effect, how well the directions agree determines how strongly that position's shift is transmitted. The authors call this weighting mechanism the gate.
What actually gets transferred to the student is the endpoint shift from Initial to Final. The intermediate Waypoint is not the thing being transferred; it serves only as evidence for gauging whether that endpoint shift is trustworthy. In other words, it is not a target to imitate.
The cost barely rises. Compared with Direct-OPD, the only extras are one frozen teacher checkpoint and one scoring pass. It does not retrain the teacher, call the grader again, or add training parameters.
Whether this method actually works has to be tested against the strongest rival, Direct-OPD. It's the difference between handing over the endpoint wholesale and handing over that same endpoint filtered by directional agreement.
| Student | Math avg (Direct-OPD → Waypoint) | Code avg (Direct-OPD → Waypoint) |
|---|---|---|
| Qwen3-8B | 53.96 → 56.14 (+2.18) | 50.59 → 53.25 (+2.66) |
| Qwen3-14B | 57.61 → 59.20 (+1.59) | 56.30 → 57.82 (+1.52) |
| Qwen3-30B-A3B | 55.96 → 57.85 (+1.89) | 56.35 → 57.54 (+1.19) |
Across three kinds of students in the two domains of math and code, Waypoint-OPD's domain average was higher in all six combinations. That means transferring the same shift selectively beats transferring it wholesale. That said, these are domain averages. It did not win on every individual benchmark.
In math, Waypoint-OPD beat Direct-OPD on 19 of 21 benchmark comparisons, losing in only two (the 14B's HMMT November 2025, 47.60→47.08, and the 30B-A3B's HMMT February 2026, 33.05→32.77). In code, it won all 15 comparisons. Still, dropping down to individual evaluations, OPD² was higher in some places (the 14B's LCB v5, and the 30B-A3B's MBPP and MBPP+). What Waypoint-OPD led on are the per-student averages.
So what about the filtering produced this difference? The authors peeled the gate off to look.
Fixing the gate at 1 — effectively turning it off (No gate) — dropped the Qwen3-8B math average from 56.14 to 53.50, a 2.64-point fall. All seven benchmarks went lower. Up to here it's as expected: stop filtering, and the score drops.
What's interesting is what came next. Instead of turning the gate off, they kept the correctly computed gate values but shuffled them onto the wrong positions within the batch (Shuffled gate). The result was 51.20 — 2.30 points lower even than turning the gate off entirely. Build a proper selection signal and then attach it to the wrong spot, and you're worse off than not selecting at all. Selecting the shift is not enough; the judgment has to land on the right position to be worth anything.
That said, you can't credit this number entirely to the gate alone. The transfer pipeline has several mechanisms layered in besides the gate, so a gate-only ablation makes it hard to fully separate the gate's pure contribution from the other components (details in the appendix). One more thing: what the gate filters for is whether the direction is consistent, not whether the direction is correct. If a shift has piled up consistently in the wrong direction from start to finish, the gate lets it through unchanged.
There were also cases where a small teacher lifted a large student above the teacher itself. The Final teacher's math average is 56.13, yet the 14B student reached 59.20 and the 30B-A3B student 57.85, surpassing the teacher (the 8B tied, essentially, at 56.14). This is because what's transferred is not the teacher's ability but the movement in preference. A large student already has its own baseline ability, and laying on top of it only the directional shift the teacher found through reinforcement learning can carry it past the small teacher's final score. What this result tells us is only that the transfer method is not capped at the teacher's ceiling. Why it exceeds by exactly this much, and how far it can exceed, is not explained by this experiment alone.
Looking at Qwen3-8B math over the same 100-update window, Waypoint-OPD was higher in 13 of the 15 comparisons after removing the shared starting value, and after step 60 it led on all three benchmarks. However, "better than Direct-OPD" and "better than the teacher" diverged by benchmark. On HMMT November 2025 it passed the teacher baseline (40.83), but on the two AIME benchmarks it stayed below the teacher baseline even while beating endpoint transfer.
When applying it, the safer stance is not to over-trust a single average number. It's good to look at the average and the individual benchmarks separately, and to also check whether scores wobble when the waypoint location is changed.
ONTLE is a service that attaches an AI PM to client companies to take repetitive work off their hands. One part of that job is classifying incoming requests, prioritizing them, and routing them. Let's follow one scene to the end.
Say a client company has been refining a request-classification model with reinforcement learning and has kept the intermediate checkpoints of that training. Without this premise, the story below doesn't hold. Now the situation arises that this model must be swapped for a larger one. There are two options: run reinforcement learning on the new model from scratch, or lift the change between before and after the old model learned its knack and feed it into the new one.
What this paper adds is not "transfer" but "transfer selectively." The process of getting skilled has mixed into it habits that were tried once early on and then dropped. Look only at the endpoints, and you can mistake even those half-abandoned traces for a knack and hand them over. Use the intermediate record to separate the shifts whose direction hardened from the ones that wavered and petered out, and you transfer only what should be transferred.
If you were to evaluate it, you'd measure the benefit along three lines. Did the classification accuracy of the swapped-in model rise? Does it drop fewer of the critical requests you can't afford to miss than before? Was this transfer cheaper than running reinforcement learning anew? Swapping models within the same client company is a fairly near-term application, but transfer between client companies, or extension to operations in general, is left as a hypothesis requiring separate validation.
This paper is designed around a setting that uses a single intermediate checkpoint, and does not address configurations using several at once. So there's no basis to claim it gets better.
This paper's experiments are limited to the two domains of math and code. Whether the same effect holds on other tasks is outside the validation scope.
The paper provides a code repository address (github.com/hhhbxloveai/WaypointOPD). It states that the implementation is built on the verl framework, uses dual-clipped PPO, and that the detailed training and evaluation settings are laid out in Appendices B and C.
This paper is not the first attempt to transfer policy shifts. What existing methods have in common is that they draw their evidence from the difference between a pair of distinct models. They have the student imitate the finished teacher's distribution (the full shape of the probabilities the model assigns to each candidate) — that's standard OPD; or they keep an update alive only when the teacher–base change and the teacher–student signal agree in direction — that's OPD²; or they lift the difference between the pre- and post-training endpoints as an improvement term and transfer it — that's Direct-OPD. Either way, they set two different models — teacher and base, teacher and student — side by side.
Where Waypoint-OPD differs is the source of the evidence. Instead of comparing against another model, it takes as evidence the two consecutive shifts the teacher itself left behind while going through reinforcement learning. The axis of comparison is not model-to-model (space) but training-stage-to-stage (time). Even OPD², the closest in that it filters by directional agreement, compares against a different model. And although it uses an intermediate checkpoint, it does not make it a target to imitate or a curriculum — which sets it apart from the progressive-distillation line that treats intermediate checkpoints as training targets.
| Item | Setting |
|---|---|
| Teacher | Three checkpoints (steps 0, 600, 1200) drawn from a single Qwen3-4B model's RL run |
| Student | Qwen3-8B, 14B, 30B-A3B (covering both dense architectures and MoE — a mixture-of-experts structure that uses only the needed parts — and sharing the teacher's tokenizer) |
| Math | Transferred with DAPO-Math-17K; evaluated on seven benchmarks including AIME, HMMT, and AMC at mean@32 |
| Code | Transferred on a Codeforces basis; evaluated on the MBPP family and LiveCodeBench v4, v5, v6 at mean@8 |
| Baselines | Base, standard OPD, OPD², Direct-OPD (the strongest rival being Direct-OPD) |
mean@32 is the average accuracy over 32 generations per problem, so it looks at ability that's less swayed by luck.
These are the mechanisms the body text said are layered in alongside the gate.