Paris is a [M] [M] city. [M] is a slot left masked because it hasn't been decided yet. How you fill these two slots determines whether the sentence becomes "a famous tourist city" or "a beautiful historic city".
One way to run a diffusion language model fast is not to compare finished sentences. Instead, you create several intermediate candidates that each fill in one more slot, and check them all at once. You line up candidates with just one more slot opened — like Paris is a famous [M] city — side by side and ask together, "would the model really fill it in this way?"
Checking them in a batch like this reduces the number of checks. But the more candidates there are, the heavier each check becomes. Decoding cost roughly splits into "number of checks × cost per check", so it's a trade: the more you cut the count, the more the per-check cost swells.
What SpecFold targets isn't the count — it's the weight of a single check. The starting point is a simple fact. The intermediate candidates being checked together differ by only one slot, so most of their internal computation overlaps. SpecFold folds that overlapping computation to reuse it, and recomputes only the slots that actually changed. It sounds simple, but telling exactly where the "changed slots" end is almost the whole technique. Let's unpack it from there.
First, grab just three terms. A token is the small unit the model processes text in, a block is a span of several such tokens grouped together, and a hidden state is the model's internal representation carrying the meaning of each position.
The common language models these days spit out tokens one by one, left to right. Diffusion language models are different. They split text into blocks and start with an entire block masked ([M]). They predict many positions simultaneously and reveal only the few they're most confident about (unmasking the [M]). They repeat filling in the remaining masked slots until the block is complete.
This parallelism of filling many slots at once is what opens the door to speed. The problem is quality. To get a usable sentence you have to run the model over the same block many times, and this repetition eats most of the inference cost. So "how do we cut the repetition" has been the long-standing homework.
Speculative decoding is a technique that pre-builds plausible future candidates and verifies them all at once. If a candidate that passed verification is actually used later, the precomputed work is reused and the checks that would otherwise lie ahead are skipped.
Back to Paris is a [M] [M] city. Call this the parent candidate. Paris is a famous [M] city, which reveals one more slot, is the child candidate. The two differ only in the one newly opened slot, "famous". Since the attention in a diffusion model looks both backward and forward (bidirectional), a single candidate is the state of the whole block, not of a single token.
So it seems like the child could just inherit the parent's computation for the parts they share. This is exactly where the trap lies. We'll get to that in a moment — first let's see why per-check cost is the problem.
Verifying multiple candidate branches at once (multi-branch) does cut the number of checks well. But every single check computes the current branch and all candidate branches in full, skipping nothing. As branches grow, the per-check cost grows right along with them. The paper directly reimplemented Spiffy (a speculative-decoding baseline that uses calibration), the representative of this approach, and measured it: as the draft budget grew, the average latency of a single check kept climbing (Fig. 3, Nemotron-Labs-Diffusion-3B, H100). This per-check cost is precisely what SpecFold aims at.
It starts from an observation. A child is almost identical to its parent and differs only at the few positions it newly revealed. When the authors solved GSM8K samples with Fast-dLLM-v2-7B and examined the hidden-state differences between parent and child across candidates, layers, positions, and steps (the relative difference of child vs. parent measured at the same layer's input, ‖H_child−H_parent‖/‖H_parent‖) (Fig. 2), they found the parent and child hidden states were mostly similar, and the large differences were clustered only near the newly revealed positions.
So can we just inherit the overlapping parts and be done? No. A change at one position doesn't stay at that position. Because attention looks both ways, the change at a single newly opened slot rides attention and spreads to neighboring slots. The paper calls this spread a "residual ripple" (Section 4.4). So "may we reuse the parent's computation" has to be judged separately per branch, per position, per layer. Prior methods didn't make this judgment at all and simply recomputed everything. SpecFold takes this judgment as its own job.
Think of it like proofreading a paragraph. Re-typesetting the whole book because you fixed one paragraph is waste. You re-set only the changed paragraph and leave the rest. But just as a longer edited paragraph pushes the layout that follows, the effect of an edit ripples to the surroundings. Pinpointing how far it pushed and reworking exactly that much — that's SpecFold.
Concretely, a position moves along the path of measure the difference → reuse the parent result → preserve its own difference → recompute if needed.
‖H_child−H_parent‖/‖H_parent‖). This measurement is cheap. It isn't running the heavy layer computation to get the child's output and then comparing — it just looks at the hidden-state difference of the two branches at the layer's input. The value used for comparison and the computation being skipped diverge here.Paris is a famous [M] city is preserved that way.To actually save with folding, the cost of selecting must itself be cheap. So the positions that exceed τ and need recomputing are gathered from all branches and compressed into a single bundle, and the heavy layers are run just once over that bundle. They aren't called repeatedly while scattered. Also, the front-portion computation cache that branches share is not copied once per branch — a single one is reused by swapping references only. Copying itself is expensive data movement. Only when the time saved by skipping exceeds the cost of this selecting and moving does it actually get faster.
One thing to be clear about: difference preservation and automatic unfolding are mechanisms meant to protect candidate diversity and output quality. That the mechanism exists and that final quality was actually maintained are two different things. The real accuracy is examined separately in the numbers below.
Implementation note (for implementers) Each layer is implemented in Triton, a language for writing GPU kernels directly. A gate kernel measures the inter-branch hidden-state difference and records the origin of folded positions, then compresses all branches' recompute positions into a single packed batch. An origin-resolution kernel follows the fold chain to connect each folded position to its actual parent state. Attention is handled with a FlashAttention-style streaming kernel, and the attention output of a folded position is re-fit (corrected) with that branch's own keys and values. The shared cache is unfolded without copying by swapping only references (a zero-copy strided view), and it also supports the attention variant where multiple query heads share keys and values (GQA).
What kind of reuse is this? In transformer inference, the paths to cut redundancy have usually been two. One is reuse along the time axis — a subsequent step inherits the previous step's computation (KV caching is the representative). The other is skipping unnecessary tokens and layers within a single pass. What SpecFold goes after has a different grain. It's the horizontal redundancy between the candidate branches evaluated side by side in the same verification step. Since the target is different, it doesn't touch how drafts are made and layers right on top of a technique like Spiffy. It's complementary, not competing.
Before reading the tables, just two things. Speed is measured in TPS (tokens generated per second — higher is faster). There are two comparison points: Vanilla, the basic block decoding that uses no speculative drafting at all, and the earlier Spiffy. The tasks split by character into code generation (HumanEval, MBPP), math reasoning (GSM8K, MATH-500), and instruction following (IFEval).
The core of the experimental design is one thing: Spiffy and SpecFold are fed the same draft graph. But even with the same draft graph, the execution path and the number of checks (NFE) don't end up perfectly identical. In fact, for Fast-dLLM-v2-7B · HumanEval in Table 1, the NFE differs — Spiffy 117, SpecFold 108 — and the accuracy diverges too. Still, the number of checks the two skip is matched to a similar level, so a substantial part of the speed difference can be explained by the per-check cost savings. Lowering per-check cost is SpecFold's core contribution, but one can't flatly claim the entire speed difference has been cleanly isolated to that alone.
First, the maximum case. The biggest gain came from the smallest model.
| Model · Task | Vanilla | Spiffy | SpecFold |
|---|---|---|---|
| Fast-dLLM-v2-1.5B · MBPP | 121 TPS | 148 TPS | 242 TPS |
SpecFold is 1.99× Vanilla and 1.64× Spiffy. The headline number comes from here. But this is the overall peak, not the expectation.
The overall trend has to be read separately. Across the same 1.5B model's five benchmarks, the range against Vanilla was wide at +54–99%. As the model grows, the gain shrinks fast: on NLD (the Nemotron-Labs-Diffusion family)-14B's two tasks it was only +10–24% over Vanilla, and against Spiffy it was essentially even at 98 vs. 102 and 110 vs. 115 TPS. The point is that the multiple grows the smaller the model, and the more you compare against a slower Vanilla.
Accuracy has a different character, so it's examined separately. For each row, which baseline the value is compared against is noted alongside.
| Model · Task | Baseline → SpecFold | Direction |
|---|---|---|
| NLD-14B · HumanEval | 71.34 → 74.39 | Up |
| NLD-14B · MATH | 84.00 → 85.60 | Up |
| Fast-dLLM-v2-7B · HumanEval | Vanilla 65.85 · Spiffy 64.63 → 61.59 | Down |
| NLD-3B · HumanEval | Spiffy 75.61 → 70.12 | Down |
On average it's comparable to the baselines. But it swings both ways by task. There are tasks where it actually rises, like NLD-14B, while on code generation (HumanEval) both 7B and 3B dropped noticeably. What matters is not the average but which side your task falls on.
Evaluation setup. Five models across two families (Fast-dLLM-v2 1.5B · 7B, NLD 3B · 8B · 14B). All zero-shot, on a single H100 80GB unless otherwise noted. Code tasks scored with EvalPlus, the rest with LM-Eval. Spiffy's calibration samples were drawn from the MBPP · MATH training splits, kept separate from the evaluation data.
Conclusion first. It only gets actually faster when the time saved by skipping exceeds the cost of selecting (picking), correcting (fitting), and moving (data movement).
Two facts were observed. One, the bigger the model (NLD-14B), the more the improvement converged to a few percent. Two, accuracy dropped on some code-generation tasks.
The reasons are left as hypotheses. What matters is that both can be settled by measurement.
Once you measure the recompute ratio during execution and the selection · correction time separately, that measurement settles whether the slowdown is because "the savings side is small" or "the cost side is large".
If you're considering adoption, conditions attach. Since the gain is sensitive to model scale and there really are code-generation tasks where accuracy dropped, per-task benchmarking is not optional. You also have to remember that the recompute threshold τ holds the speed–accuracy trade. Set τ low and you fold less — safer but slower; set it high and the opposite. You have to draw this curve for each production task and find the safe point. Finally, this acceleration is tied to custom kernels, so for a model outside the two families evaluated (Fast-dLLM-v2, NLD) or a different attention variant, you have to check kernel compatibility first.
The ONTLE stack isn't a diffusion model, and we can't just lift SpecFold's kernels. This paper is best read not as code to apply but as one question to throw: when we generate multiple candidates, how much work actually overlaps between those candidates?
Picture one scene (not that it runs this way now — this is a hypothetical example to show the structure). An ONTLE PM agent draws up a weekly plan. It lays down a shared context of last week's progress, overdue tasks, and team resources, then on top of it produces three options — an aggressive one, a safe one, a minimal-resource one — compares them, and picks one. The three options reference the same context identically. But if coded naively, each option feeds that context into the model from scratch. The reference is shared, yet the computation is not. The more candidates, the more tokens and latency swell accordingly.
The hint SpecFold gives is about order. Pay the shared computation just once and let the remaining options inherit the result. And check whether the small part that differs per option is properly reflected in the result. Skip this and the three options come to resemble one another, and the whole reason for making multiple options vanishes. It's exactly the same concern as carrying the residual ripple per branch.
Whether it worked is settled by measuring four things under the same conditions: latency per option, cost per option, candidate diversity, and the quality of the final chosen option. If folding the shared computation reduces the first two while keeping the latter two, then that fan-out (a structure branching out into several paths from one input) is worth folding. The fourth in particular — final quality — is the metric that swung by task even in this paper. The lesson that you must not skip measuring quality just because you're watching speed holds here too.
The authors checked with serialized execution and state that the implementation itself is correct (Appendix C). They explain part of the swing as the tiny numerical difference arising from computation order — i.e., floating-point noise. But it's only "part", so the drops on code-generation tasks can't all be chalked up to noise. That's why per-task empirical measurement is needed.
The paper also measures the number of forward passes through the model to solve one sample (NFE). Even feeding the same draft graph, NFE doesn't come out perfectly identical (Table 1's 7B · HumanEval is Spiffy 117 vs. SpecFold 108), but it's matched to a similar level — and if TPS still diverges despite that, a substantial part of the difference can be read as coming from the cost of a single check. This dovetails with the design intent of isolating the per-check cost.
How the recompute ratio and throughput move as you vary the draft budget and drafting strategy can be followed further in Appendix D. It's the place to gauge the main text's hypothesis that when there's less to fold, recomputation rises and the gain is shaved off.