Hand an LLM agent a long task and the model will reason, call a tool, and read the result hundreds of times over. Recomputing all the text this produces — the prompt, the messages the model writes, the output the tools return — from scratch on every step is slow and expensive. So serving systems use a KV cache: intermediate values stored so that content already read doesn't have to be recomputed. The trouble is that the longer the task runs, the more this cache balloons without end. By Tsinghua's measurements, reading this cache accounted for 82.4% of total API cost (Figure 14).
Cost isn't the only problem. There's a ceiling on how many tokens a model can handle reliably at once. Once there's too much cache to attend to, or the numbers that tell the model where each token sits (positional encoding) stretch too far apart, performance degrades. And when the task history exceeds the model's maximum input length, execution simply stops.
The standard fix so far has been to shrink the conversation history itself — trim old messages, or swap in a summary. But there's a hidden cost here. The moment you touch the front of the history, all the cache piled up behind it is invalidated. The cache is built on the premise that "the earlier tokens looked like this," so if the front changes, everything after it has to be recomputed. That's why production harnesses (the programs that manage an agent's execution) tend to hold on until they hit the limit, then summarize or delete in one big move and keep only the last few rounds. Keep only the recent ones and useful information from the middle is lost; keep a lot and you have more to recompute. It's a tradeoff between cost and information.
The Tsinghua researchers sidestepped that tradeoff. They leave the conversation history untouched and directly throw away only the cache the model attends to. And remarkably, this came out cheaper than summarizing — and more accurate.
The idea the researchers call "cache engineering" is simple. The agent and harness decide what to drop from memory, and the server executes that decision on the actual cache. Unlike past cache management, which was left to low-level signals like token position or attention scores, the decision now moves up to the side that understands the task context.
Prior work on this problem falls roughly into three camps. Text-level context engineering (summarize, trim); cache-level compression, eviction, and reuse (low-rank representations, quantization, eviction based on token position or attention importance, reuse via positional-encoding adjustment); and serving-system-level cache sharing and scheduling (prefix caching, etc.). The first two leave the decision of what to drop and reuse to low-level signals like token position or attention score, and prefix caching stands on the equation "same prefix = same cache."
Here it's worth being precise about what "throw away" means. Even if you discard the cache, the original record remains. You're just removing part of the intermediate values the model directly attends to when computing its next answer. Keeping a record and having the model reach straight into its contents are two different things.
Concretely, two new operations are defined.
There's a reason reposition is needed separately. Position numbers keep climbing with the task even as you empty the cache. No matter how much you shrink the cache with drop, the next token's position number isn't reset, so you can end up with a tiny cache but numbers shooting all the way to the model's limit. Reposition decouples the position numbers from the cache size and prevents this.
RoPE (rotary position embedding) represents a token's position by rotating its key and query by a certain angle. To shift a position you only change the rotation angle of the key; the stored values aren't subject to rotation, so they're copied as is. This lets you adjust position without recomputing the hidden state.
Drop has one notable property. Even after you discard an old memory, the cache you keep was computed while that memory was still alive (Figure 1). In other words, the influence of the discarded information is already baked into the remaining cache.
Which memories to keep came out of the data. Looking at the attention distribution, older tokens received less attention than recent ones, and tool responses received less than the model's reasoning messages at the same position. So the default policy is "tool-response rolling drop": keep all reasoning and recent tool responses, and push out the oldest tool-response cache first.
The headline first. Averaged across three models and four benchmarks, this tool-response rolling drop cut average cost by 27.7% versus conventional threshold-based summarization, and raised benchmark scores by 1.7 points on average. Rather than trading one off against the other, both cost and accuracy improved.
The comparison was actually tilted against drop. The baselines were two summarization-based methods common in agent harnesses (Standard, and Text Engineering, which only lowers the threshold to 48K), and drop's cost was billed at the cache-hit rate, deliberately charging a steep price for renumbering positions. The details — model scale, thresholds, rates — are collected in the table and the deeper-dive boxes below.
| Category | Setting |
|---|---|
| Models | MiniMax-M2.7 (230B total, 10B active) · GPT-OSS-120B (high reasoning effort) · MiroThinker-1.7-Mini (thinking mode) |
| Benchmarks | BrowseComp-Plus (BCP) · xbench-DeepSearch-2510 · GAIA-Text (103-task verified subset) · SWE-Verified |
| Metrics | pass rate · average API cost per task (cents) |
• Rates: GPT-OSS-120B at Groq rates, MiniMax-M2.7 at official rates, MiroThinker-1.7-Mini at the rate of its base model Qwen3-30B-A3B.
• Standard thresholds: GPT-OSS-120B and MiniMax-M2.7 = 96K tokens, MiroThinker-1.7-Mini = 192K tokens. Text Engineering = 48K.
• AKV policy: tool-response rolling drop keeping only the KV of the most recent 12 tool responses, with the threshold identical to each model's Standard.
• Serving efficiency metrics: request latency, TTFT (time to first token), TPOT (time per output token), input/output throughput.
• Serving experiment: 8× A800-80GB GPUs (4 prefill + 4 decode, tensor parallel 4), BF16.
For detail, the analysis running GPT-OSS-120B on the first 100 tasks of BrowseComp-Plus (4 attempts per task, up to 100 rounds) is vivid.
The observed fact is clear: at the same cost tier, drop scored higher than summarization. The authors' explanation of why is closer to a hypothesis — summarization throws away intermediate information wholesale, while with drop the remaining cache still carries the influence of the discarded information. They did not experimentally isolate and confirm the source of the performance gap. So whether this advantage holds on tasks where the answer lived inside a tool response — cases where you have to pull a past number or constraint back out verbatim — remains an open question.
This approach breaks one of serving systems' long-standing assumptions. Normally, if the earlier tokens match, the cache matches too — which is what let requests with the same prefix safely share cache. But with cache engineering, even identical text can yield different cache values depending on when, and what, you dropped. Figure 5 is the example. Both trajectory ① and ② keep the KV of B and C, but in ① A was still alive when C was computed, while in ② A was dropped before C was computed. The tokens B and C are the same, yet C's cache value differs. You can't reuse cache correctly by looking at the prefix alone; you have to track token history and the history of cache operations together. The paper calls this "history dependence."
The serving engine designed to handle this complexity is AKV. Its principle is "stateless interface, stateful serving." The agent sends only messages and cache-operation instructions, and the server takes on the tangled state management internally. Within that, it enforces three things. To avoid wrongly reusing cache with a different operation history, an extended radix tree distinguishes branches with different drop timings; to avoid carelessly clearing cache that another request might still use, drop-aware eviction delays reclamation; and when memory is tight and an evicted cache is needed again, segment-batched attention restores it in a single prefill. The complexity the serving engine shoulders grows accordingly. In exchange, attaching the AKV server made this policy up to 1.2× higher output throughput and 1.5× faster P95 request latency.
It's designed to be compatible with existing API-based serving. The messages field gives the full logical token sequence, and an optional cache field specifies drop and reposition with an "operate-after" syntax. The agent only sends messages and operation instructions, with no need to track which KV remains on the server; the server handles state management internally.
Prefill is the stage that computes all the earlier tokens at once to fill the cache. Ordinary causal prefill follows only the one-way rule that "later tokens see earlier tokens," so it can't reproduce a state in which some of the middle was dropped. Segment-batched attention chops the query at the drop points, so each segment attends only to its own tokens and "the live cache of the previous segment." Only attention is split into segments; the FFN (the neural layer that transforms each token independently) processes the whole as one batch, avoiding the cost of re-reading the weights.
Cache operations can be composed like a program. That makes recursive self-improvement (RSI) possible — the agent explores, evaluates, and revises its own cache policy on its own. The policy RSI found cut cost by a further 7.1% over a policy an expert designed by hand, on held-out tasks not used in evaluation. This 7.1% is measured against the expert-designed policy, not the same baseline as the 27.7% above.
The limitation the authors call out is exactly the history dependence we just saw. With the simplicity of "just look at the prefix" gone, the server has to track tokens and operation history together. The extended radix tree and segment-batched attention are designed to handle this, but the implementation sits on top of existing SGLang — good for reproduction and comparison, but not something you can drop straight behind any commercial API.
The place to be more careful is whether the drop policy generalizes. "Keep the most recent 12 tool responses" is a reasonable default drawn from the observation that attention looks at tool responses less. But on tasks where tool responses hold decisive facts — deep research, say, where the search results themselves are the answer — dropping could do harm instead. Tasks that require pulling the numbers or constraints from a past tool response back out verbatim are precisely the failure point. That's also likely why RSI is presented alongside, as a mechanism to re-explore the policy to fit the task. And because cost was computed from rate cards, the felt benefit can differ between environments that run GPUs directly and those billed through external APIs.
Say an ONTLE AI PM owns a project for several weeks. Every day it checks the task board, reads documents, and asks owners for status. Meanwhile the task history grows without end. It seems fine to shrink the cache — oldest first — for the repeatedly-queried board listings or full document text.
The problem is the decisive constraints mixed in among them. For instance, a single line from a meeting three weeks ago: "this feature is on hold at a specific client's request." If we were to test this study at ONTLE, the evaluation criteria shouldn't be how much cost dropped, but three things. First, does the PM pull that long-buried hold reason back out verbatim, with its basis? Second, does that hold stay distinct from decisions that were later reversed, rather than getting muddled together? Third, how much does cost actually drop while preserving this accuracy? Because the observation that tool responses get looked at less does not mean "constraints are safe to drop."
If you're on a commercial API:
If you run a serving engine yourself:
The paper sets the two against each other as alternatives. That said, since drop and reposition don't touch the text record, they aren't conceptually exclusive with text-level processing. No measurement combining the two appears within the excerpt, so we avoid asserting it.
There's no mention within the excerpt of requiring model retraining. Drop is an operation that removes cache entries from subsequent attention, and reposition is an operation that shifts only position numbers without recomputing keys. Both are operations on the cache at inference time.
Yes. The code repository is https://github.com/thu-nics/AKV, and the base serving system is SGLang.