ONTLE's AI PM carries a single client project for days at a stretch. It takes in the request, opens files, digs through the database, reads build logs, and leaves intermediate decisions along the way. All of it piles up as one long conversation. Then around day three, something odd happens. The questions are much the same, but the answers come more slowly. What used to arrive in a few seconds now visibly lags as the conversation stretches on.
That slowdown is where this post begins. SparseEngine, a recently released inference engine, aims squarely at this problem. It's a "sparse-first" engine that runs 15 different sparse attention methods (approaches that don't look at the entire past, but pick and look at only part of it) inside a single engine. The numbers about speedups of several times over catch the eye first, but the interesting part isn't actually those numbers. This engine, together with the sparse algorithms it runs, handles "how to forget"; what it leaves to us is the policy above that — which of a project's memories are worth keeping. For ONTLE, the real work starts right there. Let's follow that trail.
LLMs stopped being ask-once, answer-once tools a long time ago. These days they work as "agents" that use tools and carry on reasoning over many turns on their own. ONTLE's PM operates exactly this way. And the more it works like that, the faster the conversation history balloons, riding on tool-call results and chains of reasoning.
The key question is where the model stores that history. To keep from losing the thread as the conversation grows, the model holds on to everything exchanged so far. This "memory it holds on to" is the KV cache. It's a set of precomputed values for every past word, and all of it sits in GPU memory.
The trouble is that this memory isn't free. As the conversation gets longer, two things get worse at once. First, the memory fills up GPU memory. Once it's full, there isn't even room to start a new project. Second, every single character of an answer the model produces, it rescans that long memory from start to finish. The longer the memory, the slower each character. The ONTLE PM's answers thinning out on day three is because of that second effect. The memory that keeps it smart becomes a cost, one for one.
The sparse attention family is an attempt to cut that cost. The name sounds intimidating, but the idea is simple: shave down the cost of the memory we piled up to stay smart. Some methods keep all of the memory but look only at the part the current question needs; some throw away memory that has become useless; some store the numbers at a coarser resolution (this is called quantization) to shrink the footprint. The one thing they share is "cut the cost"; the paths to cutting it are all over the map. The first two are about choosing which tokens to look at, or discarding them outright, while compression like quantization or Palu throws away no tokens at all and merely shrinks the size of the representation. The paper, too, splits these into two separate branches: selection/eviction on one side, compression/quantization on the other.
This diversity is a headache from the engine's point of view. How the memory is represented and when it's updated differs from method to method, yet all of them have to run on the same serving infrastructure. Serving infrastructure is the shared housekeeping that lines up requests to process them (scheduling) and reuses what's already been computed (reuse). But the moment the engine hard-codes "memory is stored this way and updated this way," any method that doesn't fit that mold becomes hard to accommodate. The authors liken this to programming languages: every language makes certain computations easy at the price of making others hard. An inference engine is the same.
So the question narrows to one thing. There's the shared housekeeping everyone uses, and there's the memory control that differs per method. Where do you draw the line between them so that everything fits?
Most existing engines drew the line at "storage." They partition memory into pages, or split it across GPU and CPU, or reclaim it frugally at the token level. Each was strong for methods that matched its own storage scheme, but weak for anything outside it.
SparseEngine drew the line somewhere else: not "how you store" but "when you step in." The order in which a transformer model computes, layer by layer, doesn't change regardless of which sparse method you use. On top of that fixed execution path, the engine lays down a dense set of points where a method can step in: the stage that processes the prompt all at once (prefill), the stage that produces the answer one character at a time (decoding), the point where a layer ends, the point where a step ends. Each method hooks its code onto the points it needs, and decides for itself when to trim memory and when to discard it. The model code itself isn't touched by a single line. That's how 15 wildly different methods run inside one engine.
Here the engine draws one clear distinction. It will matter later when we read the performance numbers. "Discarding" memory and "not looking at" memory are different things. Actually emptying the storage to hand free room back to the GPU is physical eviction. Leaving the storage in place while narrowing only the range you read this time is logical selection. Freeing up space versus shrinking what you look at. The effects are nothing alike.
That's what happens inside a single conversation. On top of that, the authors added two more mechanisms to stitch requests together. Follow a day in the ONTLE PM's life and you'll see immediately why they're needed.
The PM reads a long build log. It skims hundreds of lines and distills them into one line: "the cause is a dependency version conflict." From this point on, there's little reason to keep hundreds of lines of raw log in memory. The useful conclusion has already been extracted.
With an ordinary engine, this is hard to discard. To continue the conversation on the next turn, it reuses what it computed earlier on the premise that "the earlier part is still intact" (this reuse is called prefix caching). But methods that discard memory break exactly that premise. Once you've thrown away the earlier part, reuse no longer works. So if you use a discarding method, you pay the penalty of recomputing the long history from scratch every turn.
SparseEngine's first mechanism, the Chain Cache, solves this. It bundles together the memory and state left over per session, and when the next turn comes in, it continues from what remains instead of recomputing everything. Even if some memory has already been discarded, a marker pointing to "how far the conversation has come" is kept, so the follow-up request only fills in the still-unprocessed tail. That way, when the same project thread reopens the next day, the PM doesn't reread two days of history from the top.
The second mechanism is controllable Prefix-Cache Pruning. Back to the build log. When the product designates "this stretch is less important now" and how much of it to keep, a scoring policy picks which parts within that stretch are worth keeping, and the engine empties the rest of the physical memory. But it doesn't touch the marker that tracks the flow of the conversation. So even though the raw log vanishes from memory, the conversation still continues smoothly as the same thread. One PM gets that much lighter, and the same GPU can take on more client projects at once.
A good number becomes a lie once you strip the conditions off it. What SparseEngine's evaluation gets right is precisely how it separates those conditions out.
Quality first. The authors split the baseline in two. One is each sparse method's original implementation; the other is full attention (looking at all of the past, discarding nothing). Comparing against the original implementation measures "did SparseEngine preserve the original quality while reimplementing that method"; comparing against full attention measures "how much does looking at only part of it degrade quality by itself." Two entirely different questions, kept unmixed.
Start with the reimplementation side. On the long-context benchmark LongBench, with Llama-3.1-8B, the gap between SparseEngine and the original implementation runs mostly between -0.1 and +0.8 points. Fair to call it essentially the same quality, reproduced.
The cost of looking at only part of it varies sharply with which method you pick. On the same benchmark, full attention scores 50.7, while SnapKV comes in at 50.2 and OmniKV at 50.7, nearly touching it. By contrast, StreamingLLM drops to 46.4 and Palu to 41.8, noticeably lower. So it's not "looking at only part of it degrades quality." It's "which method you pick decides everything."
And it isn't just the method you have to pick. The budget, the cap that decides how much memory to keep, also shakes the result. On the reasoning benchmark AIME, full attention for the Qwen3-4B model scores 81.7%, while SnapKV collapses to 58.3% when it keeps only 4,000 tokens' worth of memory (4k), then climbs to 80.0% at 8,000 (8k) and 85.0% at 16,000 (16k). Raising the memory it keeps even edged it slightly past full attention.
Jumping to "85.0 is clearly better than 81.7" here would be hasty. This score is the average of running 30 questions twice each, 60 runs in all. Even for the same model, there's a bit of randomness in how it picks the next character, so the score wobbles by a few points each run. Factoring in that wobble, the gap between 85.0 and 81.7 is likely within the margin of error. So the safe conclusion isn't "it got better" but "it didn't get worse." In the same table, StreamingLLM crumbles all the way to 18.3%. Which means there's a clear trade-off: the more aggressively you chase speed, the more accuracy you lose.
Now speed. Three numbers float around, and because each comes from a different experiment measuring a different effect, you can't read them bundled together. First, discarding memory physically at large batch sizes pushes aggregate throughput past 10x versus the existing engine vLLM. That's the result of using the memory freed by discarding to grow "how much you process at once." Second, under the same concurrency, the speed of producing answers goes up by 2.5x or more. That's the result of reducing the attention computation itself. Third, in an experiment that replays an entire recorded multi-turn conversation end-to-end, it's about 2x faster (up to 2.24x). But you absolutely have to view this number alongside how the experiment was run. This did not measure the completion time of an agent actually carrying out a task from start to finish; it fixed the model's output to prerecorded values and only simulated the waiting time for tools, a "trace replay" experiment. On top of that, the up-to-2.24x figure didn't come from the same conditions; it was pulled out by simultaneously raising the number of concurrent requests from 32 to 72. The three numbers point in the same direction, but what they measure differs. So reading "2x faster" as if the completion time of real agent work halves, or as a guarantee across long conversations in general, is wrong. That 2x is a value under specific conditions, arrived at by replaying a recorded conversation while also pushing up concurrency.
There's one more condition. The gain grows when the context is long. When the input is short, as in AIME, attention takes up a small share of total runtime, so no matter how much attention computation you strip out, there isn't much to strip. That's why the speedup on AIME was smaller than on the long-context benchmarks. This engine shines on tasks that decode long over a long context; otherwise, lower your expectations.
The fairness of the comparison is also worth noting. Feeding a very long input (128K) into the GLM model, the competing system Vortex ran out of memory even at batch 1 and couldn't run at all, and two other systems couldn't support the special memory-compression structure this model uses. In some settings, in other words, the competitors didn't even make it to the starting line, so you have to read the blanks in the comparison tables with that context in mind. Conversely, the authors report that even matching the competing systems on the mode they do best, SparseEngine came out ahead on the common measurement window. That's their answer to the suspicion that it might have traded away efficiency to gain generality.
That's what the engine does for you. From ONTLE's standpoint, the real work starts here.
This engine, and the sparse algorithms it runs, give you a safe way to carry out forgetting — including scoring which tokens to drop within the region and budget you hand them. What they don't judge is the business meaning behind it: whether a given memory is one you can afford to lose. The earlier build log was fine to discard once it was summarized. But not everything is that clean. A requirement the client mentioned offhandedly three days ago can turn into the grounds for today's decision. If you'd designated it "less important," the PM loses crucial context. Conversely, if you cling to everything out of caution and discard nothing, the memory headroom you were after in the first place vanishes.
So what ONTLE has to design first isn't the engine but the judgment criteria. Among a project's memories, what is core and what is disposable? Can a summarized tool result be discarded? Which user utterances get called back up several turns later? Who makes this distinction, at what point, and on what grounds? The engine won't write this judgment logic for you. That's the product's job.
An operational trap tags along, too. The Chain Cache reclaims from the least-recently-used session first when it needs room. So if a project thread that sat idle for a long time reopens, its state may already have been reclaimed, and you can pay the cost of filling it from scratch. Which projects to hold as "alive" is, in the end, something we have to decide too.
For ONTLE, the value of this paper isn't "a fast engine came out." It's that it turned a question we'd long put off, among a project's memories, which to hold and which to let go, into homework we now have to answer in code. The engine and its algorithms took on the mechanics of forgetting, so we get to focus on the policy above them: which memories are worth keeping.
The 15 sparse methods fall into four families. (1) Dynamic sparse attention: keeps all of the past but picks which parts to look at depending on the question. (2) KV eviction: actually discards memory that has become useless. (3) KV compression: shrinks memory smaller for storage. (4) KV quantization: stores the numbers at a coarse resolution. The methods that appear include SnapKV, H2O, Quest, OmniKV, DeltaKV, KIVI, StreamingLLM, Palu, and KVzip.
A few of the step-in points and callbacks: during prefill, H2O discards memory in chunks, and after the last prompt chunk, SnapKV performs selection/eviction. During decoding, Quest picks which pages to read based on the query, and OmniKV and DeltaKV reuse the indices chosen at one layer across the following layers. KIVI jointly manages a quantized region and the un-quantized residual region. All of this runs on the same model, with no method-specific dedicated model implementation.
Evaluation models: Llama-3.1-8B-Instruct, Qwen3-4B-Thinking-2507 (a GQA-family model where multiple queries share keys/values to shrink the KV), Qwen3-30B-A3B-Instruct-2507 (an MoE that only turns on the subset of experts it needs), GLM-4.7-Flash (an MLA structure that holds the KV compressed to low rank), and Qwen3.6-27B, which mixes linear attention with GQA. Hardware: H100, H20, RTX PRO 6000, 4090, 5090.
Benchmarks: LongBench V1/V2 (long context), AIME 2024 (reasoning), SWE-Bench Lite/Claw-Eval (multi-turn agents). The competing systems and their respective storage schemes: Vortex (page-level routing), SPIN (GPU/CPU tiered split), Tangram (reserves a retention quota per attention head, a head being the unit by which the model slices the context from several angles), HiSparse (keeps it in host memory and pulls only what's needed to the GPU), vToken (separates logical and physical addresses for token-level reclamation). The baseline engine is vLLM (v0.26.0).
The code is public (github.com/CURRENTF/SparseEngine). That said, the numbers in this post come from specific models, hardware, and benchmarks. Before you ship it, it's safer to re-measure quality and throughput on your own model and your actual workload.
They're different in nature. vLLM is a dense engine that doesn't assume sparsity, and its reuse rests on the traditional assumption that "the memory is still fully intact." SparseEngine didn't bolt sparsity on afterward; it laid it down as the default from the start and redesigned the whole thing from the ground up. Rather than a replacement, it's better seen as an engine with a different starting point.
The averages are trustworthy, but be careful with the details. On LongBench V2, individual subtask scores wobble more, which the authors attribute to the small sample size and the randomness in how the next character is picked. It's safer to set your quality bar at the average level, not at a single subtask.