When you put AI models from different vendors on the same team, you end up reading the same document over and over. The first model reads a long source, summarizes it, and hands it off — and the receiving model reads the whole thing again from scratch. HeteroFold removes that re-read. The intermediate computation the first model built up as it read is handed to the second model, which — after a lightweight conversion into its own form — uses it instead of rebuilding it from scratch.
Start with the results. When Llama-3.1-8B read a 32K-token context and passed it to Ministral-3-14B, the handoff was more than 10x faster than having the receiver read everything over again. On a benchmark where several agents communicate and reach decisions, it matched the accuracy of passing plain text back and forth. The key point is that this worked between models from entirely different families — different tokenizers (the way text gets split into small pieces called tokens), different layer counts.
But the most interesting part of this work isn't the speed. It's a result that cuts the other way: a method that restored the cache to look more like the original actually performed worse. That's where we'll start.
As an LLM reads a prompt, it computes a pair of vectors — a key and a value (KV) — for every token and stores them. Think of it as a scratchpad that lets the model avoid recomputing the earlier text when it later generates its answer. That scratchpad is the KV cache, and the process of sweeping through the received text once to fill it is called prefill.
The problem is that the sending side already finished exactly this prefill. We throw that scratchpad away, and the receiving side sweeps through the same text all over again. The longer the context, the faster this duplication piles up. In today's multi-agent systems that assign a different model per role (the paper points to X-MAS, which places models from different families on solving, evaluating, and synthesizing), you pay this cost again at every handoff.
Between models of the same family, you can. Across families, three things don't line up. The tokenizers differ, so the same sentence gets cut into different pieces; the layer counts differ, so it's ambiguous which layer's information should map to where; and the numerical distribution and structure of the KVs themselves differ.
Picture two translators splitting the same book between them. One wants to hand over the notes they jotted down, but their shorthand, their page-numbering scheme, and their notation are all different — the other can't use them as-is. To pass the notes along, you first have to translate them into the receiver's system.
Here a natural thought arises. Just convert the sender's scratchpad to look as close as possible to the receiver's scratchpad — the closer the numbers, the better the translation, right?
Wrong. KV Ridge, an existing method, restored the cache more faithfully than HeteroFold and yet still wrecked the receiver's attention (the mechanism that decides which token the model should focus on right now). The numbers were closer, but the outcome was worse.
Here's why. The numbers in the scratchpad aren't the goal. The real goal is the downstream computation those numbers will produce in the receiver. Focus on matching the numbers on average, and you can drift away from how the receiving model actually uses them. So instead of minimizing restoration error, HeteroFold tunes the conversion against the attention and output the receiver will produce from that cache. That change of direction is the heart of this paper.
The conversion has four steps.
Align the tokens. With different tokenizers, the same string gets cut differently. In the paper's example, one side treats "unbelievable" as a single token, while the other splits it into "unbel / iev / able." So rather than matching token indices 1:1, you align on the character boundaries both sides share. If there's no state sitting exactly on a receiver boundary, you use the state from the nearest preceding shared boundary. The important thing is not to average overlapping pieces. The model reads text in order from the start, so if you blur and average the "state up to this point," the meaning breaks.
Align the layers. The information the receiver needs is scattered across several of the sender's layers. So instead of mapping one received layer to a single sent layer, you pull in the three layers around the point matched by depth ratio. Bundling three neighboring layers made the receiver's predictions more accurate than using just one.
Match the color (Recolor). Reducing per-token error alone leaves the overall distribution misaligned and the attention skewed. So you match the mean and the spread (covariance) of the receiver's features. It's just like color-correcting a photo. You swap a differently-toned photo into the receiver's tone.
Correct against the output. Even after matching color, the attention pattern and final output still don't come out identical. You fill the remaining error against the receiver's actual query (the vector in attention that holds "what am I looking for right now"), attention, value, and output. Here, rather than overhauling everything, you use a lightweight correction table that adds only a few important directions (low-rank: fixing with just a handful of key directions instead of changing every one of the many numbers). The key is keeping the key-correction and value-correction separate: the output difference splits into a change in "where to focus" and a change in "what gets pulled in," with the former handled by keys and the latter by values.
The last step is folding. Rather than keeping the corrections above as separate pieces, you pre-fold them into a single fixed transform up front. So at runtime, each receiving layer and role just passes through one fixed transform — no extra correction network, and no re-prefill on the receiving side.
Let's be clear about one thing. Nothing in this process touches either model's weights. What gets tuned is not the model but a single conversion table that transforms the cache. That table is fit once, offline, on a small amount of data, and from then on the models are left untouched while everything just passes through the table.
Layer alignment flattens and mixes the features, so the cross-mapping holds even when the number of attention-splitting branches (heads) differs. Recolor whitens both feature spaces (removing their original scale and correlation), aligns them with Procrustes (overlapping two sets via rotation), and then re-imposes the receiver's covariance and mean. The fixed transform that folding bakes in is an affine map — a linear transform plus a shift.
The experiments freeze (fix the weights of) three models — Llama-3.1-8B, Qwen3-4B, Ministral-3-14B — and measure all six directed transfers among them. The tasks are four long-context QA sets, five short-context QA sets, and HiddenBench, where several agents communicate over repeated rounds.
First, why token alignment is decisive. Methods that force-matched token indices essentially collapsed. On two-choice questions they landed around 50% — the level you'd get by guessing — and on GSM8K, which demands numeric answers, they were very low across the board. But add token alignment (step 1 above) and much of that range snaps back to life. In other words, the first barrier to cross-family transfer was the tokenizer.
On top of that, HeteroFold pulls ahead once more. Across the four long-context sets it was the best in every direction among the cache-transfer methods, and it was best in most of the short-context cases too. That said, on long-context single-question QA there's still a gap to the baseline that passes everything as text (TextMas). The numbers show it plainly.
The speed gain grows with context length. In the same transfer direction, 4K is 3.75x over re-prefill, and at 32K it widens to 10.74x. The structure is clear: the shorter the shared context, the smaller the gain; the longer it is, the bigger.
The most interesting reversal is HiddenBench. On majority-vote accuracy after 15 rounds, HeteroFold scored 29.2% and the text-passing TextMas scored 27.7% (both starting from 11.5%). Cache transfer, which fell short of text on single questions, actually edged slightly ahead in the real multi-agent setting where agents generate and exchange new information. The earlier direction — "tune the downstream computation, not the restoration" — gets rewarded here.
What's lost is clear too. The paper itself admits two limits. The covariance matching in recolor is, in practice, an approximation because of numerical stabilization, and it didn't win on every short-context case. In some tasks on the Ministral-3-14B to Qwen3-4B direction, there are cells where existing methods remain narrowly ahead.
F1 scores on a 0–100 scale how much the answer overlaps with the correct one. On Qasper in the Qwen3-4B to Llama-3.1-8B direction, HeteroFold is 29.12, KV Ridge+TA is 16.06, and the text baseline is 44.64. HiddenBench average accuracy is 30.6% (HeteroFold) vs. 29.2% (TextMas). At the midpoint of 16K the latency gain is 8.11x, sitting between 4K (3.75x) and 32K (10.74x).
Let's narrow to a single concrete scene. A customer asks, mid-project, to move the deadline up. ONTLE's AI PM moves this request from intake to review to in-progress. At each step, the project context accumulated so far — the customer spec, summaries of attachments, decisions from earlier rounds — gets passed to the next model again, and that model reads the long context from the start all over. The longer a project runs, the bigger this repeated reading gets. It's exactly the situation HeteroFold targets.
That doesn't mean it's a part we can drop in today. HeteroFold has to reach directly into the receiving model's internals (KV, layers, query, attention) to tune its corrections. It can't be used on closed commercial API models you can't open up. ONTLE currently routes commercial APIs, so the premise doesn't hold as-is. So rather than a full rollout, the proposal is to carve out just this one segment and validate it with a small PoC.
Here's how to frame the PoC.
Set the adoption bar in advance. If quality is close enough to the text method that a human can't tell them apart, if handoff latency clearly drops on the long-context segment, and if the table-preparation cost is recouped by how often that segment is called — then we switch that segment to open-weight models and cache transfer. If even one of the three falls short, we hold, and just keep watching this technology mature alongside our open-weight operating strategy.
It's the accuracy when the agents vote independently first, before communicating with each other. Then, after exchanging private information over 15 rounds, they vote again and rise to 29.2% and 27.7%. Because the comparison starts from the same point, it shows that communication genuinely lifted performance — and that the lift held up under cache transfer too.
When the receiver takes the cache in badly, broken-format answers — invalid responses — go up. HeteroFold's invalid rate was 3.2%, lower than the text method (4.0%). By contrast, one KV Ridge variant that handed the cache over poorly saw invalids spike to 20.9%, a clear sign it had destabilized the receiving model. We read it not just as an average score but as a signal that the receiving model wasn't broken.
Transfer is per direction. Just as three models gave six directed transfers, more models mean more combinations, and each combination's table has to be fit separately offline. The more models you mix, the more the preparation cost grows accordingly.
On long context, key correction accounts for most of the gain. On some tasks, key-correction-only even scored slightly higher than the final with both key and value. It means value correction helps on average but can act as a slight offset depending on the task.