A multimodal language model (MLLM — a model that handles images and text together) can readily tell you that "the apple is to the left of the cup." But flip the viewpoint once — "So if you look from behind, which one is more to the left?" — and it starts missing. This is an easy problem for a person. Yet why the model gets it wrong was impossible to tell from its answers and behavior alone. There was no way to distinguish whether it had poor eyesight and never saw it, or whether it saw it fine but the computation in its head got tangled.
This paper cracks the model open and traces where the answer actually comes from.
On viewpoint-transformation questions (SpinBench), if you simply ask three models for the answer, accuracy is about 61%. These are two-choice questions, so random guessing already lands you at 50%. That is only marginally better than guessing.
But when you read the position coordinates out of the same model's intermediate representation and perform only the viewpoint-rotation computation externally on its behalf, accuracy jumps into the 90s. An intermediate representation is the bundle of numbers the model holds between input and output. The authors found the structure that encodes object positions inside those numbers and called it S-Space.
Only one thing changed: whether the model was left to "rotate the coordinates to match the viewpoint" itself, or whether that was handled by an external rule. The model wasn't made larger, and it wasn't given a better image. And yet the score rose this much — which means the position information needed for the answer was already inside the model, and what was missing was the computation to rotate it to the viewpoint. The paper's core finding is all contained right here. From here on, we verify "is that really so," one step at a time.
MLLMs have been expanding into physical-world tasks like scene reconstruction, navigation, and robot manipulation. But when a model gets a spatial problem wrong, the answer alone doesn't reveal where the failure occurred.
Existing approaches split into two camps. One measures spatial-reasoning scores at the output end with benchmarks (SpinBench, MMSI-Bench, CV-Bench, and others). They tell you "it's wrong" but can't pin down "where it went wrong." The other is the mechanistic-interpretability line — research that looks at how the model actually computes inside — which has prior work examining spatial encoding and representational bias.
S-Space bridges the two. It doesn't just read the internal representation; it also checks whether editing those coordinates and writing them back changes the model's judgment (causality), showing the values read out aren't mere decoration, and then turns that verified structure back around into a diagnostic tool. If a benchmark tells you "it's wrong," S-Space tells you "it saw it correctly, but went wrong in the computation that transforms it." That said, the authors frame this as a conditional claim, qualified with "to the best of our knowledge."
Underlying this is one gospel hypothesis of recent interpretability research: models tend to encode concepts "linearly" in their internal representations. That means the strength of a given concept grows and shrinks along a particular direction. The authors reasoned that an object's spatial coordinates, too, are embedded this way inside the representation of that object's name token (a token is the word-fragment unit by which a model processes text).
Back to the apple and the cup. When the model judges "is the apple to the left or the right of the cup," the internal numbers of the word-fragment "apple" contain a component relating to position mixed in. The S-Space idea is to pick out and read only the component that corresponds to position from among those numbers.
There are three axes to read out: horizontal (left–right), vertical (up–down), and distance (near–far). Read the horizontal coordinates of the apple and the cup, then subtract the cup's coordinate from the apple's. If that difference is positive, the apple is more to the right; if negative, more to the left. If the apple really is on the right, its horizontal coordinate reads larger, so subtracting yields a positive number. You don't even need to make the model speak an answer aloud — the answer falls out of the internal numbers alone.
So how do you isolate the "component corresponding to position"? The device used here is role-swapping. In the question, the apple is the "thing being asked about" and the cup is the "reference." Push the apple to the right and the "right" answer becomes more favored. But push the cup to the right by the same amount, and now "left" becomes favored. The same movement, yet the signal moves in exactly opposite directions depending on the role. Conversely, the signal coming from the object names or the phrasing of the question sits there identically regardless of role. So if you subtract the two roles' signals from each other, the commonly shared noise cancels and vanishes, and only the "position component" — the part that moved oppositely with role — remains. Subtraction is the mechanism that reveals the position information.
An axis found once this way isn't tied to a particular question. The same fixed axis fit object tokens pulled from different tasks just as well. That means S-Space isn't a product manufactured by some one question, but a structure that was in the model's representations to begin with.
Going deeper — how the three axes are found The authors didn't train the axes with labels. Instead, they differentiated the model's response. For the question "is the apple to the left or the right of the cup," they define the score gap between the two answers "left" and "right" (the logit difference — the difference in the scores the model assigns to each answer), and find the direction that raises this score gap fastest, obtained as a gradient (which way to push to make the value rise fastest). That direction is the axis. The gradients of the queried object and the reference object are subtracted off via "role antisymmetry," and noise such as object identity, phrasing, and token role is averaged away across many pairs and many examples with varied prompt templates and ordering. Note that the horizontal and vertical axes are image-plane coordinates, not true 3D world coordinates, and perspective due to distance was not separated out.
The authors split the claims that S-Space supports into three questions, and confirmed each with a different experiment. The questions stack in order. Can position be read → does the model actually use those coordinates in its judgment → can those coordinates be transformed to match the viewpoint.
Experiment details Three models were tested: Molmo2-ER, Qwen3.5-4B, and Qwen3.6-27B. Axes were found from relative-spatial questions built on COCO images, and to check generalization, coordinates were evaluated on CV-Bench, EmbSpatial, and SpatialTunnel — including distributions not used when finding the axes. Viewpoint transformation used SpinBench. The layer numbers below (layers 21, 17, 43, etc.) are the layers where each observation showed up most clearly.
Design: Check whether the coordinates read from the intermediate representation match actual object positions, and whether the answers solved from those coordinates alone are comparable to the answers the model produces directly.
Result: The coordinates read out agreed strongly, on all three axes, with actual positions and with the coordinate differences between object pairs (Spearman rank correlation, Molmo2-ER layer 21). Accuracy on spatial problems solved from coordinates alone rivaled the model's direct answers, and it held even on benchmarks not used when finding the axes. Even when the relative-comparison question was swapped for a non-spatial "color-only" question, coordinates read on the same axes propped up the spatial judgment. That means it isn't an artifact that a particular dataset or question phrasing happened to produce.
Design: Merely reading the coordinates leaves open the possibility they were "just incidentally there." So the coordinates are edited, written back, and generation continued. The image, the question, and components outside S-Space are left untouched. If the judgment follows the edits, that's evidence the model actually references those coordinates.
Result: Editing the coordinates changed the spatial judgment far more than random edits did (Molmo2-ER layer 17). When both objects were moved together by the same amount so that relative position was preserved, the answer changed relatively less (80.36% answer-retention rate per Appendix Table 7 — though about 20% still changed), and the same edit had relatively less effect on non-spatial answers like color judgments (retention rate 78.13%, with about 22% changing). It isn't "unchanged," but the effect showed up largest in spatial edits that actually shook relative position — backing the claim that the model uses S-Space causally and selectively relative to the task.
This is where results diverge. Perspective-taking questions ask which of two objects would appear more to the left, or closer to the observer, from a given viewpoint (left, right, behind). You pick one of two, so the guessing baseline is 50%.
Design: Compare the path of "read coordinates from S-Space → rotate the coordinates by the target viewpoint → extract the answer with a fixed rule" against the model answering directly (both with and without chain-of-thought). Chain-of-thought (CoT) is a method of having the model write out intermediate reasoning in text before producing an answer.
Result:
| Model | Direct answer | Chain-of-thought (CoT) | S-Space read + rotate |
|---|---|---|---|
| Molmo2-ER | 60.96% | no reasoning mode | 94.52% (+33.56%p) |
| Qwen3.5-4B | 60.96% | 62.33% (+1.37%p) | 91.10% (+30.14%p) |
| Qwen3.6-27B | 60.96% | 83.56% (+22.60%p) | 95.21% (+34.25%p) |
(The gains in parentheses are all measured against that same model's direct answer, under the same setup. Do not add up improvement figures across different models.)
Direct answers stayed only marginally above guessing. Chain-of-thought rose clearly only on the largest model, Qwen3.6-27B (+22.60%p), and stopped at +1.37%p on the smaller model. By contrast, merely reading the coordinates and rotating them externally brought all three models into the 90s. The position information needed for the answer was already inside the model. What was missing is the computation that rotates that information to the viewpoint and carries it through to the answer.
That said, don't read these 90s as "the model's transformation ability got better." The computation that rotates the coordinates is a correct transformation injected from outside by a human — an oracle (a device that externally supplies the correct working). The performance rose only as evidence that "the information was there." In fact, the authors show that even when handed the correct positional relationship in the original viewpoint as a premise, the model still frequently gets the target-viewpoint answer wrong.
That doesn't mean the internal representation is utterly unresponsive to viewpoint. But it must be made clear that what was seen here is not reasoning the model wrote out on its own. The authors ran a controlled experiment on Qwen3.6-27B (layer 43), giving it a reasoning template and filling the "original relationship" and "post-rotation relationship" slots inside it with the benchmark's correct answers (§5.2, Appendix C.5). Reading the coordinates step by step as they followed the chain-of-thought this way, they found that the relative coordinates being read drifted little by little toward the viewpoint the question demanded as the reasoning progressed. Early on, the value "read from the coordinates and then rotated externally" is closer to the right answer, but toward the later stages of reasoning the value "just read, without rotation" grows closer to the right answer. That means that when you read along a chain-of-thought with the correct answers written in, the model's internal coordinates shift toward the target viewpoint too. But this was observed under a controlled condition where the correct answer was handed over, so it can't be read as evidence that the model performs viewpoint transformation on its own.
To summarize, the three are abilities at different levels. Having the position information (Question 1), using that information in a judgment (Question 2), and transforming that information to match the viewpoint and carrying it through to the answer (Question 3) can each hold or collapse independently. These models manage the first two but collapse at the final step.
A supplementary note — direction and position can get entangled On the MMSI-Bench bearing task, the model placed "east" alongside "right" and "north" alongside "above." That means on-screen directions (right, up) and the cardinal directions are represented partly as the same features within S-Space. The authors attribute this to a bias in web training data, where bearings — as on maps — frequently come aligned to the horizontal and vertical axes.
Starting with the limitations the authors acknowledge. The horizontal and vertical axes are coordinates on the image plane, not true 3D world coordinates, and distance (perspective) was not pulled apart separately. In a case study, a phenomenon also surfaced where "socialist" projected to the left of the horizontal axis and "conservative" to the right. Interesting, but also a limitation — the model doesn't cleanly separate the physical "left/right" from the figurative uses of those words.
There are questions a reader should flag too. First, with only three verification models (two sizes of Qwen, plus Molmo2-ER), generality across architectures as a whole is still a cautious matter. Second, the concrete numbers for the coordinate correlation that underpins the "rivals the direct answer" statement aren't shown in the public excerpt, so here I simply relay the authors' qualitative description of "strong correlation." Third, the "explicit rotation" that boosted performance is closer to an oracle — a human injecting the correct transformation from outside. "How to make the model learn the rotation itself" is beyond this paper's scope, and the authors only forecast handing it off to an external tool (agentic — a design where the model calls dedicated tools to use).
Boil this paper's design principle down to one line: separate the act of reading position from the computation that follows it, and start by distinguishing where it went wrong. It's an attitude you can carry straight over to ONTLE's work of reading and manipulating screens.
Picture a scene. ONTLE's AI PM receives a screenshot of an operations dashboard and carries out the instruction "reposition the payment-conversion widget to the left of the churn-rate card." This work splits into four steps.
Up to now, when such work went wrong, we usually figured "it must not have read the screen properly" and tinkered with the prompt or bumped up the resolution. This paper says that reflexive response can have the wrong address. Because you first have to separate out whether the error occurred in step 1 extraction (perception) or step 2 computation (transformation). The way to separate them is to reproduce the two errors independently.
But distinguish between what the paper showed and what ONTLE must test as a hypothesis. The paper reports that in the three models it tested, transformation was the bottleneck. Whether that holds for our pipeline's PM model and dashboard input is still a hypothesis, and only the two measurements above will tell.
If transformation is the bottleneck as hypothesized, the design changes. Instead of shoving a sharper screenshot at it, the division of labor is: let the model read the coordinates (extraction), but compute repositioning like "to the left of the card" or "look from behind" meticulously in code and hand only the result back to the model. Prepare for the case where directions get entangled too (see the supplementary note above). Since the model has a tendency to automatically bind "east = right" and "north = up," on dashboards that mix maps, floor plans, and coordinate systems, split bearing from on-screen position in the instruction in advance. And in domains where "left/right" are also used figuratively (politics, finance, org charts), cross-verify the model's spatial judgments.