A KAIST team has released AgSpec, a framework for making coding agents faster. The idea is clean: it takes an existing acceleration technique called speculative decoding and re-carves it to fit how coding agents actually work. Only two things were reshaped — where you keep the text you're going to copy from, and how many characters you try to draft ahead at once. It doesn't build a new search engine; it sits on top of one that's already running, so it drops straight into an existing pipeline.
Let's pin down the scope first. What gets faster here is only the stretch where the model writes out its response. Not the whole task time — which includes running tests and calling tools — but the slice of that time the model spends producing characters. And this speedup happens without changing a single character of the output. You get the same answer, just faster.
Here's the magnitude. Generation throughput ran 18.0% higher on average than the previously fastest method, and up to 4.76x compared with the baseline that produces tokens (word fragments) one at a time, in order. Throughput here means the number of tokens finalized per second — in other words, how fast the text gets written out. But that 4.76x needs a caveat: it's a throughput figure measured on batches of 16 requests at a time, with prompt read-in time excluded, over the pure generation window only. It does not mean the response latency of any individual request drops 4.76x. You can only translate it into wall-clock time if you assume the same speedup applies identically to a single request.
But the part of this paper that really grabs your attention isn't the size of the multiplier. It's that the same AgSpec produced dramatically different effects depending on which engine you bolted it onto — and that the less clever the engine was to begin with, the bigger the jump. We'll come back to that once we've looked at the results.
A large language model produces tokens one at a time, in sequence. Each token pulled requires one full pass through the model. So the longer the answer, the more these sequential steps pile up, and the slower it gets.
Speculative decoding wedges a division of labor in here. A cheap "drafter" takes a guess at several of the next tokens, and the main model verifies them all at once, confirming only the ones that match. Every token it gets right saves the main model one sequential pass it would otherwise have had to run itself. The verification is designed to preserve exactly the probability distribution the main model would have produced on its own, so only tokens that pass survive. That's why, when you run it in greedy mode — always picking the single highest-probability token, as this paper does — the final output is identical to what the main model would have produced token by token. The claim of zero quality loss holds under this condition.
What AgSpec uses, specifically, is retrieval-based speculative decoding. The draft doesn't come from a separate model; it's pulled from text you already have. It searches the current prompt, the response generated so far, and a pre-built store for passages that plausibly continue the current context, and copies them verbatim. This copying works especially well on code: identifiers, snippets, and repository conventions repeat constantly, and in an agent session the same subtasks and fix loops churn out long, predictable sequences endlessly.
The trouble is that a coding agent is shaped differently from a chatbot. A chatbot is one question, one answer, done. A coding agent handles one request across multiple turns. A turn is a cycle of calling the model once and running a tool once. Roles get split — a coder that edits files, an executor that runs tests and checks the results — and model calls alternate with tool executions. Rarely does anything succeed on the first try. When it fails, the next turn rewrites much of the previous attempt and fixes it up. This exact "reuse across turns" is what no existing method pulled together in full. Some already kept past responses, and some already reached into repository files — but each picked up only a slice, and the rules that decide what to stock the store with and how many characters to draft still largely operated at the granularity of "one request," so no single method captured reuse across session, workspace, edit-shape, and draft length at once. Knitting those together is AgSpec's contribution.
For a draft to actually help, two things have to line up. First, the text you want to copy has to be in the store. Second, its shape has to match what you're about to write. No single existing method lined up both at once.
Think of it in terms of ingredients. There are three places you can pull usable text from. The ingredients already out on the counter (the prompts and responses exchanged in this session), the ingredients you've taken out of the fridge and onto the cutting board (the repository files you opened this time), and the seasonings that always sit on the shelf (a large body of pre-built reference text). Existing methods each opened part of the kitchen, but never all of it as one coherent supply. Some worked mainly from what was on the counter — the prompts and responses of this session; others did reach into the fridge for repository files, or leaned on the seasonings, which amount to someone else's pantry. What none did was keep all three — counter, fridge, and shelf — stocked together and aligned across turns, so that the output I produced in a previous turn and the file I just opened stay in the same retrieval pool as the fixed shelf. Pieces slipped through the gaps, too: a file truncated by the prompt length limit could fall out of reach, and reuse that had to carry from one turn's call into the next wasn't handled as a whole.
Shape was a problem too. The store holds text as the agent "read" it, but what a coder emits is a "patch" — an edit instruction spelling out where and how to change things. Even for the same code, the format when you read it differs from the format when you edit it. So text stored in its read-as shape ended up misaligned with the lines the coder actually writes, and matching broke.
AgSpec's reasoning goes like this. The problem isn't the algorithm for "how you match text" — it's the policy surrounding that match: what you put in the store and how many characters you draft. So it leaves the engine alone and lays two sets of policy on top.
The first tweak splits the store into three. Carrying over the ingredient analogy, they map like this.
The first two are "live" stores that come and go with the session; only the global one is fixed across sessions. This is where all three get wired into one supply — my output from a previous turn and the file I just opened now sit in the same retrieval pool as the fixed shelf, available across every call and turn.
One more device solves the shape problem. When a file is first opened, AgSpec doesn't just store the raw text it read; it also stores a copy pre-transformed into "the shape the agent will use when editing." In common edit formats, lines that stay get a leading space, and lines that get deleted get a leading minus. So AgSpec pre-builds one copy with a leading space on every line and one with a leading minus on every line. Whichever line the coder copies into the patch, it matches one of the two. The problem where matching broke because the read-shape and the edit-shape didn't line up is headed off by laying down the transformed copies in advance.
The second tweak is draft length. A draft token saves a step when accepted, but when rejected it just wastes verification. This waste grows the more requests pile in at once and the more crowded verification gets. So "how many characters to push out" is set along two axes.
One is a pre-determined ceiling. The further out a draft goes, the lower its odds of being right. It's only worth pushing out to the point where each additional character pays back the cost of verifying it. That limit is found ahead of time by replaying past work logs. Since the useful length differs by role, this ceiling is set separately per agent role.
The other is a multiplier that adjusts on the fly. Even within the same role, the useful length wavers from turn to turn, and within a turn too. So it looks at the lengths actually accepted over the last few attempts and, in real time, dials the next draft longer or shorter. The pre-measured ceiling holds the roof, and this multiplier moves within it, adapting to conditions on the ground.
Now for the part we deferred. Because AgSpec is a framework you lay on top of an engine, you can compare the same engine cleanly, "with AgSpec" versus "without." The picture that emerged here is the highlight of the paper.
On an engine that already reuses past responses and lengthens drafts to match the matched length (SuffixDecoding), bolting on AgSpec raised throughput by only 16.5%. Throughput is the number of tokens finalized per second — how fast the text gets written out. By contrast, on an engine with neither of those devices (SAM-Decoding), throughput jumped 64.6%, and the number of tokens earned per single speculation nearly doubled.
The message is clear. The less clever the engine was originally, the more potential AgSpec could draw out. A clever engine was already doing something similar on its own, leaving less room to add.
The dramatic part is that the ranking flipped. On its own, SAM-Decoding trailed SuffixDecoding across all 12 settings. But once AgSpec was bolted on, it beat SuffixDecoding in 9 of them. An originally slower engine overtook a faster one purely by swapping in two sets of policy. In one case the speedup climbed from 2.72x to 3.61x.
Even in an environment with no repository at all — where the fridge (the workspace store) sits empty — the effect held. Under that condition, the faster AgSpec variant still beat the fastest existing method by 35–145%. The credit goes to retrieving continuously from the text accumulating on the counter (the session), with length control adapting tirelessly on verification feedback.
Three evaluation models, two repository-level benchmarks (multiple agents splitting roles across many turns) and two generalization benchmarks (a no-repository environment, and one where a single agent plays every role). The measurements are throughput and accepted length over the pure response-generation window, with prompt read-in time excluded. Output quality isn't a measurement target, since under the greedy condition this evaluation uses, it produces the same output as the baseline. The comparison set was the baseline, five retrieval-based methods, and one learned-drafter method. To prevent leakage, the global store was built from a separate task log that doesn't overlap with the evaluation, and evaluation tasks were excluded from the store composition.
It makes things faster, but it isn't free. A few assumptions are baked in.
The device that pre-transforms text into edit shape leans on which edit format the scaffold — the harness wrapping the agent so it can read and edit code — uses. The format has to be known before the session for the transformed copies to be built in advance, and if the harness in use changes, the transforms have to be reworked too. The per-role ceiling is also set by replaying past work logs, which presumes an environment where you can collect those logs.
Keep in mind, too, that the size of the effect swings heavily by setting. When requests surge in all at once, verification itself gets crowded, and in some settings the gain thinned to as little as 1.08x. That means you may not get what you hoped for on large batches. The live stores grow the longer the session runs, and how their memory and retrieval overhead behaves on very long sessions is a point to examine further.
The last one is less a limitation than a restatement of character. What speculative decoding preserves is the main model's output probability distribution. Run greedy, as this paper does, and the output is identical; under random sampling, it merely follows the same distribution — there's no guarantee that individual generated texts come out identical. Either way, AgSpec only gives you the answer faster; it doesn't give you a better answer. If you want to raise the agent's success rate itself, you need a different lever.
The practical lesson of this paper is simple. The starting point for picking up speed is not "switch to a cleverer engine" but "measure what your pipeline repeats." Just changing what you feed the store and how many characters you draw back — matched to your work structure — left substantial speed on the table even with the engine untouched.
This lands especially hard for someone running many client projects at once, like ONTLE. Follow one scene. A client files a request: "fix the bugs in this repository." The coder reads files and builds a patch; the executor runs the tests. On failure, the next turn rewrites nearly the whole previous patch. On the client's screen, this stretch shows as "in progress," and as the turns pile up, the client waits. This exact repetition is where retrieval-based acceleration works best — if the store covers my output from the previous turn and the file I just opened, and the transformed copies matched to the coder's edit format are laid down.
The catch is that the grain differs by project. Which role repeats what, in which format, varies from case to case. So you have to measure, once. Only when you've gauged which turns in your pipeline rewrite what, and folded that pattern into the per-role ceilings, does this acceleration earn its keep. Cutting the client's "in progress" time, paradoxically, does not start with swapping the engine.
No. The only thing AgSpec trims is the stretch where the model writes out its response. The stretch for reading in the initial prompt was excluded from the measurement entirely. If your bottleneck is the step of digesting a long input all at once, that's not what this framework targets.
Attaching a small prediction head to the main model and training it "produces" drafts. AgSpec does the opposite — it "copies" from text you already have. So it needs no extra training and no draft-dedicated GPU. In exchange, the gain only materializes if the material to copy actually accumulates in the store and its shape matches. Repetition-heavy coding agents fit that condition well.
The pre-measured ceiling acts as a roof, so it's relatively stable, but the real-time multiplier moves within it, adapting to conditions. That said, if the role composition, the edit format, or the repetition pattern changes, the roof itself goes out of alignment. After you touch the pipeline, re-measuring the ceiling is the safer call.
It isn't a technique tied to a specific model. It was implemented and confirmed to work across several different models and several different retrieval engines. But the magnitude swings by model, batch, and workload, so the right move is to measure it small-scale on your own setup first, then judge.