SpatialScout, from researchers at Peking University and Tianjin University, is a benchmark that makes a VLM (an AI that understands images and text together) agent walk an indoor space on its own. Unlike the usual exams that pose a question and ask for the answer, here the agent moves the camera itself, looks around the room, and then submits a floor plan of "which object is where." By floor plan, the paper doesn't mean reconstructing the shape of the room — it means a set of points, each marking where the center of an object sits in 3D coordinates.
Scoring is done by an "overlap score." Every object has a ground-truth box (the space it actually occupies), and when you place a box of the same size at the coordinates the agent plotted, you measure how much it overlaps the ground-truth box. The more they overlap, the more precisely the coordinate was pinned down.
The test used 80 indoor scenes and 3,140 target objects. Even Claude Opus 5.5, the best performer, stalled at an average overlap score of 41.71% across all targets, and only 3.25% of targets had boxes that overlapped almost perfectly (IoU above 0.9). In other words, there is still a wide gap between roughly knowing where an object is and producing a floor plan that is complete and accurate down to the meter.
Until now, spatial intelligence has mostly been measured through question-and-answer (QA). You give the model a photo or a video and ask "how many objects are there," "is it on the left or the right," "how far apart are these two" — as multiple choice or a number. This approach has gaps. When the options are fixed, a model can land on the right answer by guessing or elimination. And drawing only on the common sense a language model picked up during training — "a kitchen usually has a refrigerator" — it can produce a plausible answer without actually understanding the space. In the authors' phrasing, answer accuracy verifies spatial understanding only in a "limited" way.
There is a deeper mismatch. A real agent doesn't sit still and answer from a photo it was handed. It has to look around on its own to check what's hidden, shift its viewpoint to resolve ambiguity, and merge evidence gathered across many scenes into a single, consistent 3D picture. This whole loop of observe → reason → decide, chasing its own tail, is what spatial intelligence is — and QA peeks at only one slice of its output.
Existing benchmarks (VSR, BLINK, VSI-Bench, OpenEQA, and others) also mostly live inside this QA frame and look at answer accuracy. ESI-Bench and S³-Bench let the agent interact actively, but still in a question-answering format, and Theory of Space judges map quality but deals with coarse, grid-partitioned maps. SpatialScout's distinction sits at the intersection of three things. It makes the agent output continuous 3D coordinates directly, it demands active exploration where the agent looks at the floor plan it has drawn so far and chooses where to look next, and it bundles the multi-step observation, recognition, response, and coordinate estimation into one score based on the quality of the final floor plan.
Boiled down to a sentence, the task is this. The agent is given a list stating "these are the objects in this room," one photo taken from the starting point, and the camera's position and orientation. From there it moves the camera itself to wander around and submits the 3D coordinate of each object's center, all plotted on a single map.
Picture walking into a house with the lights off, holding only a flashlight. In your hand is a note that says "three chairs, one table, two plant pots." You decide where to shine the beam, and looking at the floor plan you've drawn so far, you decide your next step — "I still haven't seen that corner." You also have to judge for yourself whether the chair you saw a moment ago, now visible from a different angle, is the same chair. Recognizing the same object across multiple viewpoints and stitching them into one is called cross-view association.
The agent has ten tools, which group into four kinds by what they do: leafing through the object list, moving or turning the camera, taking a photo, and estimating and submitting coordinates (the tool names are listed in the endnotes). The key design choice is that no tool reveals the true position or the score. The agent has to build the floor plan from the photographic evidence it gathers, and nothing else.
Among these, the triangulation tool for estimating coordinates captures the heart of the task. If you point to "this pixel and that pixel are the same spot" across two or more different photos, the tool computes the 3D coordinate of that spot. But all the tool does is the geometry. It doesn't judge whether those two pixels really point to the same part of the same object, nor how close the resulting point is to the object's center. A small computational error (residual) is no guarantee the pairing was correct, nor that the center was pinned down — because a point on an object's surface and the center of the imaginary box wrapping the object (an axis-aligned bounding box, AABB) are different points to begin with. In the end it's a spot that demands both the geometry calculation and the semantic judgment of "what am I pointing at."
There are 80 scenes, ranging from a single room up to an entire apartment. They were built in a simulator but kept the original layouts and object placements intact (the per-source scene counts are in the endnotes). Each scene contains 30 to 40 targets, and the radius over which targets are scattered runs from as narrow as 1.95m to as wide as 9.23m.
There is one device that cranks up the difficulty. About 47.42% of all targets (1,489) are cases where the scene holds several identical-looking objects (the discrepancy with the 43.3% reported in the original paper is noted in the endnotes). If four identical chairs are standing there, you have to work out "which of the four chairs is this coordinate," which makes the cross-view association mentioned earlier far trickier.
Performance is measured along two axes. One is how precisely the coordinates were pinned down (localization accuracy), the other is how completely the objects were found (coverage). When computing the overlap score, objects that were missed count as zero. So scores are reported two ways — "across all targets" (missed ones included as zero) and "across submitted targets." The gap between the two is exactly the share that omissions ate away. Center error (in meters) is reported separately, after pairing identical-looking objects to their nearest counterparts.
The average overlap score across all targets compresses "did it draw the floor plan accurately, and completely" into a single number. But that one number can't tell you where it went wrong — whether it overlooked something, or saw it but got the coordinate off. So the paper adds a separate analysis that picks apart the path (trajectory) the agent traveled, one at a time.
Seven models were evaluated (the full list is in the endnotes). No limit was placed on output or reasoning length, and each model was paired with each scene and run once.
For the average overlap score across all targets, Claude Opus 5.5 was highest at 41.71%, followed by GPT-6 Astra at 33.16%. Counting only submitted targets, these rise to 46.69% and about 41% respectively. This gap between the all-targets basis and the submitted basis is the share that unfound, omitted objects dragged down.
Turn to precision and the picture changes. The share with any overlap at all (a positive overlap score) is fairly high — Opus 77.36%, Astra 67.79%. But the share where boxes overlapped more than halfway (IoU above 0.5) drops to Opus 47.23%, Astra 35.15%, and the share that overlapped almost perfectly (IoU above 0.9) is just 3.25% even for Opus, and 1.62% for Astra. The coarse placement of "it's somewhere around there" gets done nearly half the time, but pinning the center precisely to the object's size is almost never achieved. The bottom group — Kimi-K3, Qwen3.8-Max, DeepSeek-V4.1-Flash — stalled at single-digit all-targets average overlap scores, with 0.00% above IoU 0.9.
The qualitative cases reveal a texture the numbers alone can't show. In a success case, Opus submitted all 35 targets with a center error of 0.073m and an all-targets average overlap score of 0.530. By contrast, in a failure case, DeepSeek-V4.1-Flash submitted all 40 yet landed at a center error of 1.000m and an all-targets average overlap score of just 0.034. Submitting everything doesn't make a good floor plan — which is why finding many and pinning them accurately have to be viewed together.
To see what separated the scores, you have to come back to triangulation. Estimating a coordinate correctly means clearing three stages in sequence: recognizing the same chair as the same chair across multiple viewpoints (association), pointing to the same spot on that chair identically in two photos (pixel pairing), and shifting the resulting point to sit at the object's center (center estimation). If even one stage slips, the coordinate goes badly off.
The trajectory analysis dovetails with these three stages. The authors paired Astra and DeepSeek-V4.1 — which called tools a similar number of times yet diverged in score — and dissected the paths of all 80 tasks. Astra had triangulation backing for 99.33% of its submitted objects, with geometry warnings at only 2.69%, and a median center error of 7.08cm. DeepSeek-V4.1 had triangulation backing for only 63.48%, warnings at 38.65%, and a median center error of 68.17cm. DeepSeek actually took more photos (39.63 vs. 32.41). A clue that accuracy came not from seeing more, but from walking the three stages one by one.
The failure-type analysis tells the same story. Only 6 of 40 trajectories met the success criterion ("90%+ coverage plus an average center error of 0.2m or less for the paired submitted objects"). Among the 34 failures, the most common was not finding all the objects (26 cases). Next was cases where the triangulation residual was small yet the position was badly wrong (21 cases) — the calculation was clean, but the agent had mistaken a different object for the same one, or pointed at the wrong spot, to begin with: slips in the first two stages. There were also 13 cases where the agent stopped in an under-found state with more than ten response opportunities still left on the table.
The limit the authors acknowledge themselves is cost. Claude Opus 5.5 posted the best scores but was the most expensive at an average of $28.85 per task, which, they note, is a tendency seen in other long-running agent benchmarks too. The open-source models evaluated alongside showed no clear advantage in the accuracy-versus-cost trade-off.
The more important limit, to my eye, is that the task's conditions are more generous than reality. The agent is handed the complete object list, the camera's position and orientation, and even the search radius in advance. "What to look for" and "where I'm standing and looking right now" are given for free. A real robot has to work these out on its own, so SpatialScout is closer to a controlled testbed that cleanly carves out one slice of spatial intelligence — active exploration, cross-view association, coordinate estimation — and measures it. That's not a flaw but a design choice; still, these scores shouldn't be translated directly into "the ability to figure things out alone in the real world."
The quietly hardest job in an AI PM's day (an AI acting as a project manager) is "reassembling the project's real state every morning." The tasks are in Slack, the blockers are in yesterday's call, the deploy status is in the CI logs, the owner change is in someone's passing remark. The AI PM has to actively scrape these fragments together and build a single state map of "what is where right now and what is blocked."
The overlap with SpatialScout ends exactly here — composing a state from scattered observations, and looking at the picture built so far to decide what to check next. But there's an important difference. SpatialScout's agent is handed the complete "list of objects to find." At the very least it can count what it missed. The AI PM, by contrast, starts without even knowing how many blockers there are or what it overlooked. The very act of measuring omissions is far harder at ONTLE.
So when you evaluate a product like ONTLE, you can't stop at "does it answer questions well." Just as SpatialScout reset the axis of evaluation by turning QA into state composition, ONTLE too should shift toward taking a human-curated project state as the answer key and measuring these three things.
The point that quite a few agents stopped under-found with plenty of response opportunities still left is a warning for ONTLE too. An agent that answers "what are today's blockers?" plausibly and one that combs the scattered sources to the end without missing a single item are entirely different things.