This paper introduces GAM (Grounded Action Models), a new foundation-model paradigm that hands a manipulation policy the two things it absolutely must know — "which object matters, and where is it" — not by hoping it emerges from training data, but by building it into the model architecture itself. You point at the target object with language, a point, or a box, and GAM converts that into an object-centric representation to predict actions. Across both simulation and real robots, it proved especially strong in situations where the environment shifts around (objects relocated, new targets designated, and so on).
To tell a robot "pick up the cup on the table," the robot needs to know two things. First, which object matters right now (the cup). Second, where exactly that object is (its 3D position and size). This is second nature to a human, but recent robot foundation models don't handle this "what and where" explicitly.
Today's robot foundation models are usually built on top of a backbone trained for some other purpose. A VLA (vision-language-action model), for instance, starts from a language model; a WAM (world-action model) starts from a video generation model. The trouble is that these backbones never directly require metric grounding — quantitative positional information about exactly where an object sits in meters. So the robot ends up having to pick up that sense implicitly, as a side effect, from robot demonstration data.
Think of it this way: it's like handing someone who has only ever read a stack of recipes and telling them "grab that pot." The recipes never spell out where or how to grip the pot, so this person has to feel their way to it through endless practice. And the moment the kitchen layout changes even slightly — move the pot to a different spot — they're lost. That's exactly why existing models are so brittle to changes in the environment.
GAM's central bet is simple: "Before you predict any action, grab hold of the target object in 3D first." That act of grabbing hold is precisely 3D grounding — connecting language and instructions to a concrete target in actual space.
Here's how it works, in everyday terms.
One more important design choice is that GAM can be used in two ways. It can operate autonomously on its own, or it can plug in as a low-level controller directed by a high-level planner. In the latter case, the high-level planner tells GAM "do this next" using those varied input modes — language, points, boxes — and that lets the system handle long-horizon tasks and memory-dependent tasks (ones you can only solve by remembering what you did earlier). Picture a field commander (the planner) and a skilled worker (GAM) dividing the labor.
The abstract reports results from two simulators and two real robots. Every figure below is taken straight from the abstract and nothing more.
| Benchmark | GAM | Baseline | Key condition |
|---|---|---|---|
| RoboTwin 2.0 (average over 50 tasks) | 55.3% | Spatial Forcing 52.0% | Action policy trained only on clean-scene demonstrations |
| RoboTwin 2.0 (scene randomization) | 47.6% | Abot-M0 30.4% | Scene randomization condition |
| LIBERO-PRO (average over 16 perturbation settings) | 61% | π_{0.5} 53% | Largest gains when targets are relocated or newly designated |
| Real bimanual robot YAM (visual shift) | 17 of 20 successes | π_{0.5} 4 of 20 | Visual shift condition |
| Franka + Molmo2 planner combination | ID 64.7% / OOD 49.8% steps completed | — | Long-horizon, memory-dependent tasks |
A few things here are worth dwelling on.
First, on RoboTwin 2.0, GAM was trained only on clean-scene demonstrations, yet it still scored 47.6% under randomly cluttered conditions. That's a wide margin over the baseline (Abot-M0 at 30.4%). It amounts to "learned only in a tidy kitchen, yet does well in a messy one" — evidence that 3D grounding confers robustness to environmental change.
Second, the note that gains on LIBERO-PRO were "largest when targets were relocated or newly designated" captures GAM's identity especially well. The design of explicitly holding onto the target object is exactly what pays off in those situations.
Third, the gap on the real bimanual robot under visual shift — 17 of 20 successes versus 4 of 20 for π_{0.5} — suggests the effect carries beyond simulation into the physical world.
That said, it's worth being clear that all of these numbers are scores under specific conditions of specific benchmarks. Average success rates like 55.3% or 61% are still only a bit above half, which means not that robot manipulation is "solved" but that it is "meaningfully better than before."
Even from the abstract alone, there are points worth naming honestly.
This piece is grounded in the paper's abstract and does not cover dataset scale, training cost, or detailed architectural figures, none of which appear in the abstract.
ONTLE is a SaaS that dispatches AI project managers to lighten the load of repetitive human work. This is a paper about robot arms, but here is what we think is worth carrying over to an actual product. The point below isn't that robot experimental results can be transplanted straight into software agents — it's an analogy at the level of design principles.
First, don't leave "what and where" to implicit learning — pin it down in structure. Just as GAM fixes the target object in an explicit representation, an AI PM agent should explicitly hold onto "what it is currently dealing with" (which project, task, or document) at every step. Once that slips, the agent does the wrong thing the moment the environment (the conversation context) shifts even a little. It's the same principle behind why, in RAG (retrieval-augmented generation), "grounding that pins down the exact target document" governs answer quality.
Second, diversity in how targets get designated. Just as GAM accepts language, points, and boxes alike, a practical agent reduces instruction ambiguity when it converges everything — whether the user instructs in words, clicks a specific item to point at it, or designates a range — into the same internal representation.
Third, division of labor between planner and executor. GAM's "high-level planner plus low-level controller" structure is structurally similar to the "PM agent (plan) plus worker agent (execute)" division ONTLE is aiming for. The abstract's note that GAM handles long-horizon and memory-dependent tasks through this separation is a reference point that resonates with where we're heading in our agent orchestration design. That said, robot manipulation results don't empirically validate a software agent design.
Fourth, robustness comes from representation, not from sheer data volume. The result that it "learned only on clean scenes yet held up on cluttered ones" suggests that designing a representation that properly grabs the target makes for a system that is resilient to change — more so than blindly pouring in more data. In the kind of workplace automation that fills in for hands you can't hire, an approach that fixes "what matters" in structure may be more economical than piling up data for every exceptional case.
Based on the abstract, it proposes a new foundation-model paradigm with 3D grounding built in rather than claiming to replace them. It showed higher success rates than the comparison models (Spatial Forcing, Abot-M0, π_{0.5}, and so on) across several benchmarks, but with absolute success rates around 55–61%, it's too early to talk about a full replacement.
It means connecting an instruction — language, a point, or a box — to the actual target object in 3D space. GAM packs not only the target's visual features but also metric geometry such as its actual size and position into the object-centric representation, grabbing hold of "what" and "where" together.
According to the abstract, GAM scored 47.6% under scene randomization (versus 30.4% for the baseline) even though it was trained only on clean-scene demonstrations, with the largest gains when targets were relocated or newly designated. The interpretation is that explicitly holding onto the target object lets it keep its focus even as the surroundings shift.
Yes. Four of them — "make the target explicit in structure," "converge varied instruction modes into a single representation," "divide labor between planner and executor," and "get robustness from representation rather than data volume" — aren't a direct transfer of robot experiments to software, but they are analogies worth borrowing at the level of design principles.