When you take a language model and train it one more time on a specific task — a step the field calls "post-training," meaning the extra work of adapting a finished model to a particular use — the mark of that training shows up even in decisions that have nothing to do with the task. This paper takes that mark — a phenomenon the authors call a "behavioral shadow" — and uses it to transfer part of a teacher model's coding ability to a student model, using nothing but ordinary word choices that don't contain a single line of code. No access to the teacher's internal parameters, no access to the internal scores the model assigns to each word, no reference code. Just the single "next word" the teacher picked for each prompt.
Fine-tune a model on coding data and it writes better code. That much is common sense. But is the influence of that training really sealed inside "coding"? That question is where the authors start. When they ask a code-trained model to fill in the blank in a short story, even that choice — one with nothing to do with code — shifts in subtle ways.
The backdrop here is a line of prior work called "subliminal learning": the finding that a teacher's tendencies or preferences can pass to a student through training data that looks completely unrelated on the surface. But earlier work mostly focused on tendencies and preferences, and it required large volumes of long outputs generated by the teacher.
The gap this paper digs into is twofold. First, observability — can you glimpse a post-training run that was done privately, without looking at the changed parameters or the task responses, using only the individual choices the model makes on unrelated prompts? Second, capability transfer — if you train on those choices, does part of the task skill the teacher gained carry over to another model? The crux is whether you can transfer a capability, not a tendency, and do it with an extremely small amount of information.
The method is called ATD, Active Taskless Distillation. "Distillation" refers to the technique of moving the ability of a larger, more advanced model (the teacher) into a smaller, follower model (the student).
Start with the setup. There is one public base model. Someone privately post-trained it on coding data to create a "teacher." We have a "student" that starts from the same base model, and the teacher is a black box we cannot see inside — all we can get from it is a single next word per prompt.
The clever part is where you aim the questions. Using the public base model, the authors pick out only the prompts where two ordinary words are almost exactly tied in their chance of being chosen (a near-tie). Picture a scale balanced perfectly level. In that state, the lightest touch of a finger tips it to one side. Post-training is exactly that faint finger. By seeing which word the teacher picked at a balance point, that single choice becomes a signal of which direction the training pushed.
The paper's actual example makes it intuitive. In some unrelated context, the public model prefers "tie" over "jacket" by a hair (normalized probability around 0.51 — essentially a coin flip). But the code-trained teacher picks "jacket." That one flip becomes a training pair. The very fact that the choice flipped in a context with no code at all is a "1-bit signal" of which way the teacher was trained. One bit is a single yes/no, the smallest unit of information.
The procedure has three steps.
The strongest evidence came from coding. Code either runs correctly or it doesn't when you execute it, so you can't lift the score by mere imitation.
The experiment used one small public model, Qwen2.5-1.5B, as both the base and the student's starting point. The teacher is the same model with one extra round of coding training on top.[^setup] The student learned only from the words the teacher picked on unrelated prompts — one per prompt, 5,664 in total. The result: on HumanEval+, an exam that measures coding skill, it scored 5.34 percentage points (pp) higher than the control.
What the "control" is turns out to be decisive. The control the authors used keeps the exact same set of words but scrambles the pairing between prompts and words (in the paper's terms, an "exact disruptive permutation"). In other words, the information lives not in which words appeared but in which word was attached to which prompt — the pairing. The 5,664 responses used for training were checked beforehand and confirmed to contain no code, math, benchmark, or task terminology.
The transfer didn't stop at coding. Across six multiple-choice exams spanning scientific knowledge, commonsense reasoning, and reading comprehension, scores rose broadly, and improvements showed up on average across other Qwen generations and sizes, as well as the Llama family. The results held whether the student was touched up lightly (LoRA) or fully re-tuned. The magnitude, though, wasn't as dramatic as with coding.
A few properties are especially interesting.
The authors draw the boundaries clearly themselves. The transfer survived only when the teacher and student started from the same public ancestor. Where that premise broke — an "incompatible ancestor" — it did not hold. And this channel is narrow-bandwidth — scraping one bit per prompt, it could hardly be otherwise.
One more thing worth stating honestly: a large jump in the teacher's task skill does not guarantee transfer (Section 5.5 of the paper). A big improvement doesn't necessarily turn into a shadow that carries over.
Still, don't hang everything on the single number 5.34pp. That figure is, at the end of the day, the result of the Qwen2.5-1.5B coding experiment measured against one particular control. The "average improvement" seen on other tasks and models is only a qualitative trend in the same direction, not a gain of the same size. Folding the two together as if they were one result would be an overstatement.
We deal every day with managing clients' private models wired in through APIs. Not knowing how a client's model was trained, throwing questions at it from the outside and consuming only the answers — that's our default situation. For us, this research cuts two ways.
Defense comes first. This paper shows that private training leaves a mark even on outputs that look unrelated, and that capability can leak out through that mark. Opening a model up via API means that, even without handing over answers directly, it keeps shedding crumbs — the word it chose. It's a warning that when we expose a client's fine-tuned model to the outside inside our product, we shouldn't calculate the upper bound of leakage from "the complete responses we serve" alone. Of course, as the authors note, the channel is narrow and shared ancestry is a prerequisite — so it's more accurate to read this not as an immediate real-world threat but as a result that redraws how we look at the threat.
There's also a use side. The client models we operate sometimes change their answers one day out of the blue, because of a vendor update or a configuration change. Performance may slip below what it used to be (regression), or the grain may shift bit by bit (drift). Right now we mostly notice these changes only after benchmark scores drop. Watching whether choices flip on balanced prompts is worth exploring as a light drift sensor for our monitoring (a probe — a detector that peeks at the state). But one caveat is decisive: the paper's early-signal diagnosis leaned on the teacher's internal per-word scores (logits), which a word-only API does not expose — so we cannot assume a probe that sees only the chosen word would catch a change before benchmark scores waver. This isn't a feature the paper proposed directly; it's our own reading, and a cautious one.
Boiled down to a single line, the key lesson is this. A model's behavior does not sit politely inside the walls of its task. Whether you're trying to transfer a capability or to protect one, you can't use "looks unrelated" as grounds for safety.
Classic distillation has the student match the teacher's prediction distribution (the scores — logits — assigned to each word), and language-model distillation usually uses the responses, solutions, or rationales the teacher produced. ATD uses neither the teacher's probabilities, nor its task responses, nor its parameters. It observes only the single word the teacher picked on an unrelated prompt.
Because code won't run unless it's correct. Unlike other subjective tasks, you can't raise a coding score by "imitation" alone. So if the coding score went up, something actually useful carried over. And since it was compared against a control with the identical set of words but a scrambled prompt-word pairing, the source of the effect is cleanly isolated: it's the pairing itself — "which prompt it was attached to" — not "which words appeared."
No. Where the teacher and student don't share the same public ancestor — an "incompatible ancestor" — the transfer did not survive, and the channel itself is narrow. Also, a large jump in the teacher's task skill does not guarantee transfer (Section 5.5 of the paper).
At a point where the two words have nearly equal probability, even a tiny shift in preference flips the choice. That makes a faint change like post-training easy to surface. It's just like a level scale, where a single finger decides which way it tips.
[^setup]: The training setup that produced the teacher is a rank-16 code-DPO LoRA on top of the base model. The transfer results held with both this approach (LoRA) and full-parameter fine-tuning.