Multimodal LLMs — models that handle images and video alongside text — don't read an image directly. They first chop it into visual tokens (the units an image is broken into so the model can read it) before processing. The more tokens, the higher the compute and memory cost, yet until now models have converted every part of an image into tokens at the same uniform density.
VisionWeave, a joint effort from Alibaba's Qwen team, the University of Science and Technology of China, and others, is a study that cuts the token count by letting the model decide on its own which regions need close inspection (fine-grained) and which can be read loosely (coarse-grained). This ability is learned through self-distillation, using the original, un-reduced model as the teacher. Self-distillation uses the original model's outputs as the supervision signal instead of creating separate ground-truth labels — it does not mean no external training data is used. The actual training drew on 777K samples pulled from LLaVA-OneVision, LLaVA-Video-178K, and UGround (per the abstract, §2.4, and Table 2).
Starting with the results: on the Qwen3.8-27B backbone across eight benchmarks, VisionWeave cut visual tokens by 43.0% on average while retaining 98.9% of the original performance. Conventional methods that prune tokens at a fixed 50% ratio landed around 88% of original performance — but the two figures aren't directly comparable, since the reduction conditions differ (VisionWeave's 43.0% average vs. the comparison group's fixed 50%). The fair, reduction-matched comparison is covered separately below.
Information in an image isn't evenly distributed. The tiny print in a document or the location of a button needs to be read in fine detail, but a flat region like a solid-color background or the sky won't change the answer no matter how loosely it's read. It's like reading a newspaper: you trace the article text letter by letter but let your eyes skim over the white margins. Using the same resolution everywhere amounts to holding a magnifying glass up even to those margins.
Existing approaches to reducing this waste fell into three camps, by how they decide what to keep.
The paper distills their weaknesses into four points. ① A training-inference mismatch, where behavior at training time diverges from behavior at inference. ② Information needed by a given task gets thrown away. ③ A fixed compression ratio that can't adapt to each input. ④ Designs that extract attention weights or prune tokens inside the LLM, making them hard to bolt onto real serving engines like SGLang or vLLM. When the authors evaluated FastV and VisionZip themselves, scores dropped sharply on information-dense documents and on grounding tasks that require pinpointing exact locations.
ViCO is the closest in spirit to VisionWeave, but its tile granularity is a drawback. It doesn't fit well with native-resolution models like Qwen or Kimi, which process images at their original size, and detail-rich regions and flat backgrounds are often mixed within a single tile — so one resolution per tile can't subdivide finely enough.
VisionWeave is built from two mechanisms and one training process. The authors call this ability "elastic visual representation weaving." Like weaving dense and loose threads into a single fabric, it interlaces tokens of two resolutions region by region.
(1) A mechanism that summarizes four tokens into one — the gated spatial pooler. The image is grouped into 2×2 blocks. A block can either stay as four fine tokens or be compressed into a single coarse token. The simplest compression is to average the four tokens, and VisionWeave is designed to start training from this average-pooling state, then gradually refine — through learning — where within a block to place more weight.
(2) A mechanism that chooses what's okay to summarize — the granularity router. For each block, it assigns a "probability that this can be made coarse." If that probability exceeds a threshold (0.5 by default), the block uses a single coarse token; if not, it keeps the four fine tokens.
Apply the two mechanisms to a screenshot of a tax invoice and it looks like this: the small-print block holding the amount is left with its four fine tokens by the router, while an empty background block is summarized into one coarse token by the pooler. The selected tokens are then mixed into a single sequence and handed to the LLM.
(3) The process of learning to match the original's answer — self-distillation. The teacher is the original model's ordinary processing with no token reduction; the student is the reduced version with woven-together tokens. The student is tuned to produce the same answer as the teacher. Here two objectives interlock. Because pressure to reduce tokens and pressure to preserve the original's answer are applied together, the router learns not to cut just anywhere, but to pick "the places where cutting doesn't change the answer."
So the three approaches diverge in behavior. Pruning removes tokens, so detail can't be recovered later even when needed; VisionWeave, instead of discarding, summarizes into a coarse representation and keeps it. The ViCO style selects a resolution per tile, but since the unit is a tile, it can't subdivide any finer.
The pooler refines each patch feature with LayerNorm, has a gate score which channels to weight, and then a value net refines the features once more. Initializing the final layer to zero recovers exact average pooling, so training can start from a safe baseline. The coarse token is placed at the geometric center of the original four tokens; within the MRoPE coordinate system, positional interpolation assigns it the block-center coordinates, so the coarse and fine tokens share the same coordinate frame.
The router pulls features from several depths of the ViT (low-level detail from early layers plus semantics from later layers), keeps one query per block, and alternately passes them through local blocks (seeing only their own block) and global blocks (seeing the whole frame) via cross-attention, after which a linear head produces the probability.
Training has three stages. ① Turn routing off, leave every block coarse, and train only the pooler. ② Freeze the pooler and train only the router, replacing the hard choice with a differentiable soft mixture, and use a balance loss borrowed from MoE to push the share of coarse blocks toward a target (80%). ③ Jointly fine-tune the pooler and the LLM on the hard-mixture mixed-granularity sequence actually used at inference. Distillation is entirely off-policy and uses forward KL over the teacher's top vocab. They report trying an on-policy variant of stage 3 but seeing no consistent gain, so they chose off-policy.
Two models were evaluated: Qwen3.5-4B and Qwen3.8-27B. The baselines are VisionZip (merging family) and FastV (pruning family); to measure fairly inside SGLang, they used a FastV variant (FastV†) that selects tokens right after the vision encoder rather than inside the LLM.
The benchmarks span a deliberately wide range of domains. Natural images (RealWorldQA), documents and infographics (DocVQA, InfoVQA), GUI grounding (ScreenSpotV2), visual hallucination (HallusionBench), reading text in video (VideoOCR), and video understanding (Video-MME, LongVideoBench) — eight in all. Scores are on a 0–100 scale; DocVQA and InfoVQA use ANLS, HallusionBench uses aAcc, and the rest use each benchmark's own metric.
Pinning down what this table does and doesn't measure keeps you from conflating the numbers that follow. What this experiment measures is "how much does accuracy drop when you reduce tokens" — the efficiency-quality trade-off. By putting detail-critical documents and grounding alongside redundancy-heavy video in one table, it looks at whether the reduction ratio shifts with the nature of the input. Conversely, this table alone can't tell you a particular service's real cost or latency. That's a different axis, handled by the SGLang serving experiment later.
There's one more design choice for fairness. The claim "VisionWeave uses fewer tokens and still scores higher" can be unfair because the reduction ratios themselves differ, so the authors also added a comparison that grants FastV† and VisionZip the exact reduction ratio VisionWeave observed (Appendix E.3).
On Qwen3.8-27B with a 512-token-per-image budget and a 0.5 threshold, it cut the average token count across the eight benchmarks by 43.0% and retained 98.9% of the original performance (average loss of 1.08%). On the smaller Qwen3.5-4B, the loss was 0.90%. Reading these numbers through two questions brings out the key point.
First, how much was cut on which inputs. The reduction ratio varied by task.
| Input type | Task | Token reduction | Quality |
|---|---|---|---|
| Documents & infographics | InfoVQA / DocVQA | 16.5% / 27.3% | DocVQA loss 1.23% |
| GUI grounding | ScreenSpotV2 | 55.1% | loss 3.17% |
| Video | VideoOCR, Video-MME, LongVideoBench | 49.6–54.2% | loss under 0.30% |
On detail-critical documents it reduced conservatively, and on redundancy-heavy video it reduced aggressively. This is the training structure seen earlier — a router that learned to pick "places where cutting doesn't change the answer" — showing up as different choices depending on the nature of the input.
Second, whether a quality gap remains even at the same reduction ratio. It does. At the 50% fixed-reduction target, the average loss on Qwen3.8-27B was 11.82% for FastV† and 12.46% for VisionZip, and VisionZip in particular dropped 47.88% on ScreenSpotV2. On that same task, VisionWeave actually saved 55.1% while holding the loss to 3.17%. Even in the reduction-matched comparison (Appendix E.3), VisionWeave's average of 76.73 beat FastV† and VisionZip's 70.10 and 70.33. It doesn't win by using less — it wins by choosing where to cut more wisely even when using the same amount.
They also checked whether the reduction translates into real speed. On Qwen3.8-27B served with SGLang, end-to-end throughput reached 2.30×, average TTFT (time to first token) fell by 54.4%, and average TPOT (time per output token) fell by 60.6%. The fact that adjusting the threshold at inference time changes the compression strength (§3.5) is another knob you can use in operation.
In experiments that varied the resolution and frame count, it was also consistently better than plain downsampling. In one example on LongVideoBench, VisionWeave capped at 256 frames used fewer tokens than the original model capped at 128 frames while scoring 1.20 points higher. That's a signal that for the same budget, it's better to pack in more frames and compress each one elastically.
Editor's note: The paper's "Limitations and Future Works" (Appendix A) is outside the excerpted range, so the authors' own list of acknowledged limitations couldn't be carried over. Below are the structural limits that can be inferred from the body and abstract.
First, training cost is high. Instilling this ability in Qwen3.8-27B took over 30K A100 GPU-hours. That's a high wall for a small team to reproduce as-is on an arbitrary model.
Second, the compression unit is tied to the 2×2 block. Because the minimum unit of a coarse token is this block, subdividing any finer than a block is out of the design's scope.
Third, there's the shift in input distribution. The stage-2 balance loss targets an 80% share of coarse blocks, and before stage 3 the router bias is separately calibrated offline toward a 50% token-reduction target — so if the inputs a service handles differ greatly from those at calibration time, how much recalibration is needed, and how far simply turning the threshold at inference covers it, is something to verify on real traffic.
Fourth, because training is self-distillation, there's a risk that the original's blind spots carry straight over. It's safest to recast this as a verification question — "in the small-number regions the original model already misreads, does error increase further after compression?" — and measure it before adoption.
ONTLE is an AI-PM-dispatch service that takes repetitive work off the plates of small businesses and solo founders.
Consider one work scene. Every morning, an AI PM reads tax-invoice screenshots from vendors, a shopping-mall admin dashboard, and a delivery-status table through a multimodal model to extract amounts, account numbers, and statuses. The cost structure here is tricky. The small numbers on a tax invoice or the position of an admin button go wrong with even a little blurring, yet the same screen also holds huge margins and repeating table headers. Apply fixed compression and the numbers come out wrong; apply none and you spend full tokens even on the margins, leaving cost unchanged.
So when ONTLE considers adopting a technique like this, the criterion isn't the size of the numbers but the order of operations. First we compare whether errors in extracting amounts and account numbers increase; only within the range that holds that acceptable quality do we evaluate processing time and cost. The paper's numbers are results on specific benchmarks and a specific backbone, so they can't be transplanted directly — but the property of "conservative on documents, aggressive on redundancy-heavy inputs" can serve, within this review frame, as a checkpoint: "first diagnose which side our workload falls on."
According to the paper, the authors state they plan to release the SGLang code and other details at github.com/FFY0/VisionWeave. The specific timeline and scope of the release aren't confirmed within the excerpted range.
A coarse token sits at the geometric center of four patches — a half-integer coordinate. SGLang's RoPE cache uses integer indices and can't hold this directly, so the authors doubled every position coordinate and halved the rotation frequency to preserve the rotation phase, reusing the existing kernel as-is. In effect, they themselves filled the very gap they pointed to — that pruning-style methods are hard to apply to real serving engines.