If you run AI models on your own servers, there's a common mistake about what eats the most GPU memory. People usually picture the model itself, but in practice the "scratchpad that remembers everyone you're currently talking to" can be the heavier burden. As users pile up, that stack of scratchpads swells until, at some point, it overtakes the model itself.
STEPQuant compresses that scratchpad down to 6 bits. In an environment where you self-host a linear-attention model and serve many concurrent users, it noticeably reduces memory while keeping accuracy essentially identical to FP32 (32-bit, near-lossless). The largest saving the paper reports is 68.7%, but that figure comes from a specific setup — Qwen run with 4-bit weights (W4) at batch size 512 — and it's measured against the sum of weights, the reserved state pool, and the KV cache, not total server memory (it's the appendix's accounting basis, and excludes some components and allocation overhead). In other words, it works under defined conditions — but let's save that for later.
The trick isn't compressing harder. It's handling errors selectively. Where prior quantization mostly focused on the "spatial" side of error — "which positions in the state matter" — STEPQuant adds a "temporal" axis: "how long does this error stick around." That's the method's core contribution, and this post follows that temporal axis.
The softmax attention everyone uses today grows its "KV cache" of past history as the word fragments (tokens) exchanged pile up. Linear attention sidesteps this by cramming all of the past into a recurrent state — a single, fixed-size summary scratchpad. No matter how long the context gets, the scratchpad stays the same size. The update scheme that erases and overwrites a bit of this scratchpad at each step is called the gated Delta-rule, and models that mix this scheme with ordinary attention — hybrids like Qwen3.8-27B (hereafter Qwen; architecture name GDN) and Kimi-Linear-48B-A3B-Instruct (hereafter Kimi; architecture name KDA) — have been showing up a lot lately.
But that "fixed size" advantage turns into a different bottleneck at the serving stage. Every concurrent request has to carry its own separate scratchpad. Add in the spare slots a serving system pre-allocates, and even with short contexts, more concurrent requests mean the "state pool" holding all those scratchpads swells right along with them. In fact, under the official SGLang deployment settings, Qwen's FP32 state pool overtakes the model itself (BF16 weights) at around 70 concurrent requests.
The way to lighten the scratchpad is quantization — a compression method that represents numbers with fewer bits by rounding them more coarsely. Shrink a value written in 32 bits down to 6 bits and the footprint drops sharply, but a tiny error creeps in. Usually the model tolerates a bit of accumulated error. Quantize the state uniformly, though, and accuracy collapses. Qwen's long-generation average fell from 80.60% at FP32 to 45.04% under 6-bit uniform quantization — a drop of about 44%.
Why the state is so touchy becomes clear when you compare it to the weights. Model weights are like a photo taken once and done. Quantize them once at load time and they stay put for the whole inference run — that rounding error is fixed in a single frame. It still tints every activation and output that follows, but it's a one-time error: no new error gets injected as generation proceeds. The recurrent state, by contrast, is more like a portrait you keep repainting over, layer by layer. Now imagine skimping on paint and only ever brushing on faint, washed-out color. One faint stroke is fine. But stack another faint stroke on top, and another, and eventually the whole portrait's color fades away. At each decoding step that produces one character of the answer, the state is updated, then re-quantized and stored — so a fresh rounding error is injected at every step, feeds back as input to the next step, and compounds. That recursive re-quantization loop — not error propagation as such, which weights have too — is exactly what "quantization error accumulates" means.
The starting point is a single observation: not all errors in the state carry the same weight. Their influence splits along two different axes — how long they persist (time), and how important a position they sit in (space).
Errors that survive longer cause bigger problems. So you concentrate your defense on the places where these "zombie errors" are likely to form. That idea is Lifetime-aware Bit Allocation.
Some slots in the state get overwritten and vanish almost immediately; others are preserved for a long time. An error lodged in a long-lived slot leaves its stubborn mark across many decoding steps. The math says the same thing. When the state updates, an error pointing in roughly the same direction as the incoming information gets shaved off quickly, but an off-direction, wrong-way error only fades slowly, relying solely on the "forget ratio" (the gate) (Proposition 1). The longer a slot holds onto memory, the more a misaligned error lingers like a zombie. The measurements draw the same picture: when you deliberately compress memory slots coarsely, the slots that hold on longest accumulated the most error (Fig. 1(b)).
So within a fixed average bit budget, you funnel more bits to the units whose errors are both large and long-lived. Each unit's bit width is chosen by a score that combines its reconstruction distortion with how long, on average, that slot holds onto memory. The unit differs by architecture: in Qwen it's a whole memory slot (head), in Kimi it's a single key row. The handful of high-risk units that simply won't compress are kept in FP16 as-is, protected as pivots.
Even an error of the same size hits the output differently depending on which position in the state it sits in. So each row gets a row-impact score for how much it contributes to the final output error. Verifying this is simple: splitting key rows by impact and dropping just one group at a time to 4 bits, the higher-impact groups made perplexity (how confused the model is about the next character — lower is better) worse.
One more observation attaches here. In the state matrix, the conspicuously large values show up along both rows and columns (the horizontal and vertical lines of a spreadsheet) at the same time. Matching the scale on just one axis can't capture both. This is where it diverges from prior LLM quantization, which generally assumes the large values cluster on a single axis and targets only that one. The pattern held across all 2048 decoding steps. That leads to Key-Row-Aware Dual-Axis Fitting, which gives rows and columns each their own separate scale.
For both models, 6-bit STEPQuant differed from essentially-lossless FP32 only in the second decimal place. Compress uniformly at the same 6 bits, though, and Qwen collapses from 80.60% to 45.04% (about a 44% drop) and Kimi from 61.52% to 45.70% (about a 26% drop) — effectively becoming different models. The difference came down to whether you simply compress, or compress selectively by deciding where to spend the bits.
| Setting (long-generation avg) | STEPQuant | Uniform quant | FP32 |
|---|---|---|---|
| 6-bit · Qwen | 80.59% | INT6 45.04% | 80.60% |
| 6-bit · Kimi | 61.47% | INT6 45.70% | 61.52% |
| 4-bit · Qwen | 80.51% | INT8 71.86% | 80.60% |
| 4-bit · Kimi | 58.52% | INT8 56.02% | 61.52% |
Even on the more aggressive 4-bit budget, STEPQuant beats uniform INT8, which uses twice the bits. That said, Kimi 4-bit still trails FP32 by about 3 points — meaning it isn't a clean sweep across every combination. On short-generation tasks, 4-bit STEPQuant lagged FP32 by just 0.15 points on Qwen and 0.25 on Kimi, and even in a production-style setup with weights squeezed to 4-bit AWQ, 6-bit STEPQuant came within 0.05 points on Qwen and 0.33 on Kimi. The more you compress the weights, the larger a share the state takes up in memory — so state compression is, if anything, more clearly worth it in that combination.
Efficiency checks out from several angles too. 6-bit STEPQuant compresses the recurrent state by 5.03x on Qwen and 5.08x on Kimi — but this compression ratio (the 5.08x on Kimi in particular) is a theoretical figure based on the storage format, not a measured allocation (Appendix D). Under the same memory accounting (summing weights, the reserved state pool, and KV cache, excluding some components and allocation overhead), Kimi's memory dropped by 53.7%, and state updates ran up to 2.91x faster.
How was this possible? Because the two axes complement each other and create synergy. In the ablation study (Qwen) that peeled off components one by one, the combined time-and-space STEPQuant hit 84.72% at 4-bit and 84.51% at 6-bit — beating both space-only (73.95% / 80.19%) and time-only (12.87% / 75.63%), converging on FP32 (84.61%). Neither axis alone is enough; the time axis in particular actually collapses when used on its own. The pivot effect is big too. Keeping just 1.39% of Qwen's heads in FP16 raised the 4-bit three-task average by 6.70 points and the 6-bit AIME by 14.16 points. In other words, protecting a mere ~1% of the whole buys a large jump in accuracy.
ONTLE is a service that dispatches an AI project manager (PM) to take repetitive work off the plates of small businesses and solo founders. Dozens of customers each keep their own AI PM open all day, each carrying its own records and conversation context. Self-host a model to cut costs, and the GPU hog is easily not the weights but the state pool maintained separately for every concurrent session. Picture having to wonder "do we need to buy another GPU?" every time you onboard a new customer.
If you were ONTLE's CTO, you'd have to ask three questions before deciding to adopt STEPQuant. First, is our service really straining because of the state pool? You have to separate out whether serving is bottlenecked on the weights or on the state pool and measure it. If concurrency is low and load is light, it's low priority. Second, is the model we run even built to take this technique? It has to be in the gated Delta-rule family (GDN/KDA), and the validation is limited to just two models, Qwen and Kimi. Third, has our traffic grown large enough to offset the burden of re-calibrating every time we swap models?
When all three answers are "yes," STEPQuant raises the ceiling on "how many customers can ride on a single GPU." Part of the old choice to add a GPU per new customer gets a chance to be absorbed into a compression setting instead. But this isn't a one-line config change — it comes bundled with the calibration and kernel work described below.
The experiment's core question is singular: if you hold the weights fixed and change only the state quantization, how much does quality suffer? Every design choice is built to isolate and measure exactly that. Evaluation ran Qwen and Kimi on SGLang atop four NVIDIA A800 GPUs, with FP32 state as the baseline and uniform INT4/6/8 as the comparison points. The tasks were split into 7 long-generation and 6 short-generation ones, to separately test error accumulation on long outputs and damage on short outputs. The short-generation tasks were also scored on the actual generated answers, because the situation STEPQuant targets is real serving where the model pulls out one character at a time on its own. Running the weights in two settings, BF16 and 4-bit AWQ, was to check that state compression still holds when paired with weight compression, and calibration used WikiText-2 (32 segments, 2048 tokens each).
What gets fixed once, offline before the model runs, is not the dual-axis scales but the bit allocation, the FP16 pivots, and the row-impact scores. The dual-axis scales themselves are recomputed to fit the state as it's updated at every decoding step. The row scale weighs that row's current magnitude against its impact score, and the column scale is tuned to penalize reconstruction error on high-impact rows more heavily. In effect, large-value rows get a wider range and important rows get finer resolution. That said, this scale fitting and compressed storage run on a separate CUDA stream so they don't block the decoding path, and the process of unpacking, updating, and emitting the state during decoding is fused into a single SGLang kernel. Since much of the performance gain comes from this fused kernel, if you use a different serving stack you need to factor in the kernel porting cost up front.
How it's positioned is worth noting too. Prior mainstream quantization (SmoothQuant, OmniQuant, etc.) assumed large values cluster on one axis and targeted only that axis. Q-Mamba (DSQ) for Mamba is the closest neighbor, but pulling the row-impact score into the scale selection is what made the difference. Even taking the space fitting alone, it hits 73.95% at 4-bit, far ahead of the 7.64% you get from porting DSQ over as-is. Mixed-precision bit allocation itself isn't new, but adding the "memory lifespan" time axis on top of it is STEPQuant's contribution. It even got close to FP32 in the sub-8-bit regime that the contemporaneous work DAMP had declared difficult.
The remaining gaps are clear too. There's the Kimi 4-bit weakness, and losses remain depending on the architecture-and-bit-budget combination. Since the evaluation is limited to two gated Delta-rule hybrids, it's hard to conclude from this paper alone whether the same structure holds for other linear-attention variants or pure SSMs. And because calibration used WikiText-2, how well the pivot selection generalizes to differently-distributed domains like code or math is something to watch further. The design of packing compressed values tightly into memory complicates the implementation, and the burden of re-calibrating on every model swap remains.
There's one bonus finding. Dropping the state via uniform quantization doesn't just collapse accuracy — it produces over-generation, where output drags on all the way to the length cap. Under 4-bit uniform quantization, Kimi pushed out an average of 63,400 tokens on AIME and 64,220 on HMMT, nearly hitting the 65,536-token cap while accuracy sat near zero. It's a collapse into verbose circling without ever finding the answer — and STEPQuant steers clear of this trap.
In theory, slightly. But for Qwen, the share kept in FP16 is only 1.39% of the heads, so the impact on the overall compression ratio is negligible. Since protecting that small share buys a large gain in accuracy, on balance keeping it is the better deal.
If you can barely give up any quality, 6-bit is the safe bet. For both models the gap from FP32 stays in the second decimal place. If you have to push compression harder, 4-bit still beats uniform INT8 — but Kimi 4-bit in particular means accepting about a 3-point loss. It's best to measure a small sample per model and task first before deciding.
The state-pool bottleneck comes from the number of concurrent requests, not context length. The scratchpad size itself is fixed whether the context is long or short, so even short conversations will swell the state pool if there are many concurrent connections. Conversely, if connections are sparse, this technique is low priority even with long contexts.
It's not a replacement. KV cache quantization targets the softmax-attention cache that grows in proportion to token count, while STEPQuant targets the fixed-size recurrent state. Since the compression targets themselves are different, in a hybrid model the two techniques aim at memory regions that don't overlap.