A single black shadow floats on the left side of the screen. On the right sits a 3D object, tilted at some arbitrary angle. The job is simple: rotate the object bit by bit until the shadow it casts lines up exactly with the target shadow on the left.
It doesn't happen in one move. You look at the current state, rotate one notch, look again at how it changed, and rotate once more. You repeat this until it matches the target. This "act, observe the result, correct the next action" repeating structure is called a closed loop. Whether it's a robot arm or a software agent manipulating a screen, every real-world agent lives inside one.
RoTA-Bench, from researchers at Xiamen University and Nanyang Technological University, measures exactly how far today's vision-language models (VLMs — models that process images and text together) get inside this closed loop. The result tilts hard to one side. By their count, the best-performing model evaluated, GPT-5.6-Sol, solved 17 out of 100 tasks. Twenty undergraduates ran through the same test set four times over and solved all of them. What this benchmark targets is a single distinction: the ability to recognize an image and the ability to turn what you see into precise 3D manipulation are different problems.
VLMs read pictures pretty well these days. They can name the object in a photo and tell you whether it's on the left or the right. But nearly all of these abilities have been measured in a "look at one fixed image and answer once" format. The paper calls this a static benchmark. There's no action, and no feedback loop that feeds results back in.
Interactive benchmarks aren't entirely absent. But as the paper lays out, each has a gap. Web browsing and OS manipulation run only inside a 2D flat screen. Maze navigation and sliding-tile puzzles are about finding a path on a grid aligned to rows and columns, so there's no change in the rotational pose of an object at all. That left an empty seat for measuring the one ability of "reading 3D orientation with your eyes and translating it into action." The paper also draws a line against approaches that feed 3D into the model directly (injecting a separate coordinate system or 3D representation). RoTA-Bench looks at whether a model can reach 3D understanding from a 2D screen alone, without such aids.
Underneath this sits one design judgment. The researchers restricted physical manipulation to rotation only. There are two reasons. First, turning an object sideways changes its outline dramatically and hides parts behind it — it becomes a visual problem that only the eyes can solve, hard to pin down in words. Second, using rotation alone keeps messy physics variables like pushing, friction, and grip out of the picture, so the ability to read spatial orientation can be isolated cleanly.
Picture the task as a single scene: a target silhouette (a black shadow) is given, and next to it sits a 3D object rotated to some awkward angle. The model picks from a fixed set of rotation moves to turn the object little by little until the two shadows overlap.
The set of available moves comes out to a clean 24. Three camera axes (pitch, nodding up and down; yaw, shaking left and right; roll, spinning within the screen plane) × two directions (forward/reverse) × four magnitudes (from a large angle down to a fine, tiny one). A single STOP, meaning "done," is tacked on.
At each step, the model holds four inputs in hand:
The degree of overlap is measured by projection IoU (Intersection over Union — the overlapping area of the two shadows divided by their combined area; the closer to 1, the more perfectly they overlap). The moment IoU crosses a threshold (0.90, per the figures and text) is a success.
The researchers didn't use common 3D CAD models. Familiar shapes trigger the prior knowledge a model memorized during training ("a chair looks like this"), letting it fall back on memory instead of looking and reasoning. Instead, they took 2D vector icons from an open-source icon library (Game-icons) and turned them into unfamiliar 3D solids by extruding them in the depth direction while confining them inside a sphere (sphere-bounded extrusion). This method comes with two mathematical guarantees. The canonical front view reproduces the original icon's fine outline exactly, and since every point of the object falls within the radius, no matter how you rotate it nothing gets clipped off the screen (the camera scale is globally fixed).
The other is reversible initialization. Starting from the correct pose, they randomly stack reversible rotations to form the initial pose, but forbid the immediate inverse of the previous move to prevent idle back-and-forth. Since simply retracing the stacked order in reverse always reaches the answer, every generated task is structurally guaranteed to be solvable. They made 674 this way and fixed a 100-item test set by stratified sampling on reference-path length (sampling to spread the length distribution evenly). The remaining 574 were left for training and research.
The breadth of the comparison is what gives this paper its persuasive force. Nine VLMs, two agent frameworks (ReAct, Magentic), 20 undergraduates, two random baselines, and two greedy baselines that get to peek at the ground truth were all run under the same rules, the same action space, the same decision budget, and the same success criterion. The setup is below.
| Item | Details |
|---|---|
| 9 VLMs | GPT-5.6-Sol·Terra·Luna, GPT-6-Astra, Gemini 3.6 Flash, Claude Opus 5, Claude Sonnet 5, Qwen 3.7 Plus, SpatioLM-Understanding-InternVL3.5 |
| Common settings | temperature 0, decision budget 48 steps |
| Commercial model calls | API, total cost $1,482.07 |
| Local deployment | SpatioLM = 8× A100 (40GB) |
| Human evaluation | 20 undergraduates, each handling a block of 20, running the full test set four times (400 runs total) → all 400 succeeded |
The evaluation metrics split into two layers. The outer layer is task performance — success rate, average final IoU, average number of rotation steps. An episode ends when it succeeds, when STOP is pressed, when it hits the 48-decision budget, or when it stalls without improving.
The inner layer is what separates out the cause of failure. The researchers compute the true 3D pose error, kept from the model, as ground-truth data on the side. And they break each rotation into direction (which axis, which sign) and angle (step size) and score them separately. The Effective Direction Rate is the fraction of moves that chose the correct direction to reduce pose error, while the optimal-angle rate, angle gap, and angle loss measure how precisely the angle was chosen, given that the direction was correct.
Let's make one thing clear up front. IoU only measures apparent overlap; it does not measure whether the 3D poses actually match. Because of symmetry and orthographic projection, different 3D poses can cast the same shadow. So pose error is used not as a success criterion but strictly as a diagnostic. Apparent overlap and true pose are not the same thing — this distinction is the key to reading the results that follow.
The top success rate was GPT-5.6-Sol's 17.0%. Next came Gemini 3.6 Flash and GPT-6-Astra at 10.0% each. Claude Sonnet 5, Claude Opus 5, and Qwen 3.7 Plus had 0% success rates, with final IoUs barely above uniform random (0.473). SpatioLM was more extreme still. It suffered a policy collapse in which 91.12% of its actions piled onto the single move YAW_POS 10°, landing at 0% success and a final IoU of 0.463 — below random. Humans, by contrast, achieved 100.0% success and an average final IoU of 0.923 in an average of 16.22 steps. According to Figure 3, the human trajectories cross the 0.90 threshold, while the models' trajectories flatten out early, somewhere around 0.50–0.63.
The next analysis splits whether this failure arises in direction selection or angle control. The two greedy baselines are the doorway. Both are cheating baselines that can compare all 24 follow-up moves in advance using the ground-truth simulator, but they look at different things. IoU-greedy picks the move that maximizes the next step's apparent overlap; pose-greedy picks the move that minimizes true 3D pose error. The results diverged. IoU-greedy, chasing only apparent overlap, managed just 39.0% success (final IoU 0.725), whereas pose-greedy, which knows the true pose, solved everything at 100.0% success (0.924, 8.32 steps). In other words, a strategy of merely growing the overlap right in front of you stalls on many tasks, while access to the true pose solves them all.
Why does chasing overlap alone stall? Take one case. You make a move that improves the overlap, but that move swings the object into a different pose. The shadow looks closer to the target, yet the 3D pose has actually drifted further away. The immediate overlap is a gain, but to reach the target afterward you now have to undo the pose gap you just opened, so the path only gets more tangled. In fact, in the same situation (matched states), GPT-5.6-Sol makes these "overlap improves but pose alignment worsens" moves more often than humans do (19.4% vs 10.9%). It's the same story as IoU-greedy getting hooked by the surface metric.
The direction-and-angle decomposition paints the same picture. GPT-5.6-Sol's effective direction selection rate is 58.75% — far short of humans (83.91%) and barely above uniform random (53.78%). Even when the direction was chosen correctly, the average deviation between the chosen angle and the optimal angle was 6.49°, more than double humans' (2.99°), producing an average angle loss of 2.25° per step (humans 0.96°). On the other hand, the amount it looked around (surface coverage) was 79.58% for GPT-5.6-Sol, close to humans' 83.07%. That means the cause of failure is not "it didn't look around enough."
The last piece is history. These are the results of an input-ablation experiment run on 30 tasks with GPT-5.6-Sol. The baseline with all inputs provided succeeds 5/30 (16.67%), final IoU 0.5656. Remove the auxiliary depth view from here, and the success rate drops to 3.33% and IoU to 0.5490. Remove the interaction history, and success vanishes to 0/30 and IoU falls all the way to 0.4822, pressed right up against uniform random (0.4808). The angle distribution tells you even more. With history removed, 96.76% of the model's actions bunch onto the single largest (30°) step. The correlation coefficient between prior IoU and chosen angle size is -0.472 with the full input (fine-tuning with smaller angles as it gets closer) but collapses to -0.242 with history removed. Without the memory of past attempts, the model can't do "we're almost there, so let's turn gently."
Put together, the message is one thing: the ability to recognize an image and the ability to turn visual evidence into precise 3D control are different axes, and current frontier models show a large gap from humans on the latter.
Start with the lines the authors drew themselves. The task is restricted to pure rotation (by design, translation and friction were deliberately excluded), so these results shouldn't be read as directly extending to general manipulation mixing grasping, pushing, and translation. Pose error is a diagnostic, not a success criterion, and because of symmetry, cases that "succeed but don't have zero pose error" exist as a matter of course. The ablation experiment was run on a 30-item subset rather than the full 100, and they presented the statistical grounds (Spearman correlation, bootstrap confidence intervals) that this subset's distribution matches the whole well.
The open question starts from the fact that pose-greedy hit 100%. This cheating baseline was handed three things at once. (1) the ability to know the current state — it knows the true 3D pose; (2) the ability to predict action outcomes — it simulates all 24 follow-up moves in advance; and (3) the ability to choose the next action — it picks the move that most reduces pose error. So it's not "it's easy if you just have pose information," but rather "it becomes easy when, on top of pose information, you're also handed the means to compare outcomes in advance." That makes the real-world battleground clear: how much of (1) and (2) a model can reconstruct from visual feedback alone, and which parts you supplement with external tools (scaffolding like explicit pose estimation or calls to a 3D simulator). The 17% success rate is just a snapshot for a particular model generation, prompt, and agent configuration; it shifts depending on how far you can push that supplementation.
Where this result lands most directly for ONTLE is the structure. A situation where an AI PM operates an ops dashboard on its own — say, looking at a screenshot of a campaign budget-allocation screen and dragging a slider to match a target split — is a closed loop of look → drag → look at the changed screen again → correct. What we share with RoTA-Bench is this iterative observe-and-act structure, not the numbers of the 3D puzzle themselves. So the following are written not as conclusions but as hypotheses for us to test.
By Table 1, it was not a clear improvement. ReAct had the highest final IoU numerically at 0.628, but a 16.0% success rate, and Magentic came in at 2.0%. Neither beat the single model GPT-5.6-Sol's 17.0%. In short, merely wrapping the reasoning process in a framework did not unclog the bottleneck of "choosing direction and angle precisely."
It's not that simple. On top of knowing the true pose, pose-greedy was also handed the ability to simulate all 24 follow-up moves in advance and pick the best among them. In practice, the question remains of how far the model does these three things on its own (grasping the current state, predicting outcomes, choosing an action) and from where you supplement with external tools.