This research evaluates the general-purpose model GPT-6-Astra on tasks of "embodied intelligence" — the ability to sense your surroundings with a body and physically move to get things done. In this field, the received wisdom has been to train a dedicated, task-specific model for each task. GPT-6-Astra, by contrast, posts unprecedented scores with a single general-purpose model. A Joint Research Team including SenseNova ran the model across 12 benchmarks under a unified set of rules to measure the ceiling of its ability, then put it on 8 kinds of real robots to see how far those scores actually carry over into the real world. The one-line summary: in the test environment it largely passes the "can it solve this?" bar, but inside the time, cost, and safety constraints of the field, "is it actually usable?" is still a no. The authors call the former threshold embodied competence and the latter practical embodied intelligence. Even if you never build a robot, anyone trying to put a general-purpose model to work on real tasks runs into the same gap — it works in the demo, but in the field the time, cost, and verification stand in the way.
To get a robot to "hang up a cup" or "fold a cloth," the standard approach has been to train a policy (a scheme that takes observations and decides what action to take) on robot data. The VLA (vision-language-action model) is the canonical example. Such policies can generalize across many tasks too — π0.5, for instance, is a VLA trained on diverse data and designed to operate even in home environments it never saw during training. The common catch, though, is that extending to a new robot or distribution usually means collecting a commensurate amount of data and retraining.
GPT-6-Astra unsettles this premise. Without touching its weights, a single general-purpose model handles everything from tabletop robot arms to humanoids, mobile platforms, and intricate multi-jointed hands. Instead of training a new dedicated model, it layers "how to operate it" on top of a foundation model that is already smart. That raises three questions. Where is this model's true ceiling? How much does benchmark success carry over to real robots? And if competence like this has arrived, what should future benchmarks be measuring? The report digs into these three questions in turn.
Two devices sit at the heart of the evaluation.
The first is in-context learning (ICL). This is a way to adapt the model to a task without changing its weights, by feeding reference material into the input (the context) on the fly. GPT-6-Astra looks over the very demonstration data that prior specialized policies used for training — before the test — and writes itself a single "knowledge package." It's like having a new hire write the job's operating manual themselves, then letting them consult only that manual on the floor. This package is frozen before the test begins and handed, unchanged, to every episode that performs the task once. What it holds is roughly a task description, motion examples (poses and movements extracted from demonstrations), reusable helper code, and procedures for checking and undoing failures. The model chooses what to include on its own, and the authors note that these same categories showed up across benchmarks even when the benchmarks differed.
The second is execution budget expansion. If performing one action is one step, then on top of each benchmark's official step limit (the standard budget, 1×), the model is allowed up to 10× the steps. GPT-6-Astra notices its own failed moves, revises its strategy, and retries — but a tight step limit cuts the episode off before that recovery can finish. The hypothesis is that given more time, the process of correcting a failure into a success can run to completion.
Separately, a RoboDojo experiment tests collaboration through a harness (the execution structure that actually runs the model). A slow but careful System 2 (GPT-6-Astra) decides "what to do," while a fast System 1 controller (here, a π0.5 fine-tuned on RoboDojo) handles "how to actually move the joints." The key is closed-loop collaboration: System 1 proposes an action, System 2 reviews and revises it before executing, then observes the result and folds it back into the next decision. Propose → review → execute → observe result → next decision, one lap at a time. This collaboration structure is not the common framework across the whole evaluation, though — the main results, including the LIBERO family, come from a setup where this VLA controller is turned off and GPT-6-Astra controls directly.
The 12 settings group by character into five strands: tabletop manipulation, dexterous hand manipulation, humanoid loco-manipulation (handling objects while walking), long-horizon household activity, and vision-language navigation (moving by following instructions). The individual benchmark names are laid out in the table below. GPT-6-Astra's weights are fixed throughout every evaluation.
| Task type | Individual benchmarks |
|---|---|
| Tabletop manipulation | RoboDojo, RoboLab, LIBERO-Long++·LIBERO-Ultra-Long, LIBERO-PRO |
| Dexterous hand manipulation | Bench2Dex, DexJoCo |
| Humanoid loco-manipulation | SIMPLE |
| Long-horizon household activity | BEHAVIOR |
| Vision-language navigation | VLN-CE(R2R·RxR), LHPR-VLN |
The evaluation rules separate "preparation" from "test." In the preparation stage the ICL knowledge package is built and frozen; each test episode starts fresh in a new session, receiving only the task instruction, the frozen package, and the permitted observation/action interface. The interaction record within a single episode is kept, so the model can diagnose mistakes and attempt recovery, but no information carries across episodes. The extended budget is a continuation of the same episode rather than a reset, so it drags along all accumulated consequences too (spilled water, dropped parts).
The comparison baselines are each setting's best public score (SOTA) and best public agentic technique (Agentic SOTA), with a SOTA cutoff of October 12, 2026. What this design measures is "the ceiling of ability when demonstrations are provided and ample time is given." What it conspicuously cannot measure is just as clear: generality in the absence of demonstrations, and the real wall-clock time reasoning takes (latency), which does not count against the step budget. As we'll see later, these two are what trip things up in the field.
Even on the standard (1×) budget with only ICL turned on, GPT-6-Astra matched or exceeded the best public score in most settings, and on LIBERO-Ultra-Long and DexJoCo it nearly doubled the best public success rate.
The effect of demonstrations. The SIMPLE experiment isolates ICL's effect. On the four shared tasks, task-level ICL lifted the standard-budget success rate from 70.8% to 90.8% and also shortened the length of successful episodes. That gain, however, depends on how closely the training data covers the test task. When the distributions are close — as with the LIBERO family, SIMPLE, and VLN-CE — it approaches saturation; the farther apart they are, the weaker it gets.
The effect of retries. Raising the budget to 10× roughly tripled the success rate on RoboDojo, which demands precise bimanual manipulation, and completed about three-quarters of the episodes that couldn't finish at 1× (Bench2Dex also more than doubled).
Tasks that stay hard even with more time. LHPR-VLN, a long-horizon navigation task, rose only slightly with the extended budget and stayed the lowest of all settings. The report's takeaway is narrower than "unsolvable": for some failures more time converts them into successes, while for others — LHPR-VLN among them — extra budget brings only limited improvement.
Why extra attempts work on some failures and are useless on others shows up in RoboDojo's mug-hanging. The model sees the scene through a camera and reads the view where the handle and the peg overlap as a "hung" signal. When the signal says the task isn't done, it realigns its pose, but because it misread success in the first place, it keeps repeating the same misjudgment. That's as far as the report's observations go.
From here one interpretation follows (this is the author's inference, not the report's observation). For a retry to turn into recovery, three things have to be present together. First, the failure has to be identifiable — as in the mug case, if the success check is off, more time only piles up the same failure. Second, there has to be a corrective action still available. Third, there has to be a state left that can be corrected — once the water is already spilled or the part already dropped, there's no place to undo it. RoboDojo's precision group recovered through retries because all three were in place, and the three residual failures we'll see below are each cases where one of these conditions breaks.
The final success rates with both ICL and the extended budget turned on split by task character as follows.
| Setting | ICL + extended budget (10×) success rate |
|---|---|
| LIBERO family · SIMPLE | over 95% (near saturation) |
| VLN-CE | about 90% |
| DexJoCo | 87.3% |
| RoboDojo | 81.9% |
The validation on 8 real robots aims directly at the "reality" in the title. GPT-6-Astra worked across a variety of robot bodies, but its success rate hinged heavily on the task. It did well at tabletop grasping (picking up objects), while deformable-object manipulation like handling cloth, precise contact, and long-horizon tasks remained hard. It showed active vision — moving the camera itself to see what it needed — and in some cases pulled off humanoid manipulation by having the slow System 2 collaborate directly with a high-frequency locomotion controller (called "System 0" here). In reality, though, safety constraints, mechanical limits, and real-time demands mean the strategy of "buying success rate by spending more steps" doesn't work, and the successful demonstrations that simulation ICL took for granted are either expensive to collect or simply nonexistent.
The biggest constraint is cost. The authors note that about 98% of the tokens consumed are cached input tokens and use "1 million tokens ≈ $1" as a rule of thumb. The problem isn't the token price — it's time. Even LIBERO-Long++, which already exceeds 97% success on the standard budget, costs about 1 million tokens and nearly an hour of reasoning and tool use per episode. In other words, even at a high success rate, finishing one job takes close to an hour. LIBERO-Ultra-Long jumps to about 9 million tokens and nearly two hours, and with the extended budget one RoboDojo episode averages over three hours.
This cost is tied directly to harness design. Closed-loop collaboration lifts performance far above solo execution or an open loop, but the more often System 2 is called, the more tokens and time balloon together. The authors report that reviewing longer action sequences at once improves this trade-off, and that combining the closed loop with ICL yields the highest performance but also the highest cost. The way actions are represented matters too. Controlling each joint angle directly, one by one, hurts performance; it's better to tell the end-effector (the hand or gripper at the tip of the arm) "what to do" in semantic units. Adaptive action chunking — bundling several moves to review at once — raises performance while cutting unnecessary model calls. In short, the core of the design is to open up an action space that matches the model's reasoning ability while leaving precise execution to a lower-level controller, and to trade accuracy against cost through review frequency. With the same brain, how you build the harness dictates performance — even the latest agentic technique (Agentic SOTA) using the same GPT-6-Astra is outpaced by this approach.
The residual failures the authors acknowledge directly come in three strands, and each lines up with one of the "three retry conditions" breaking.
First, it can't verify physical success. It notices that a task isn't done but can't pinpoint which physical condition isn't met, so its corrections don't converge. On the Bench2Dex jigsaw puzzle, it reports that four pieces form a square and yet, seeing the episode isn't over, keeps adjusting the gaps until it actually pulls the pieces apart again.
Second, it can't recover from irreversible failures. Spilling water or dropping a part early on tears down every subsequent subtask wholesale.
Third, it can't preserve progress state in long-horizon tasks. Progress that a later step depends on — a "lid left open" or an "object set down" — vanishes, and the task stalls.
Transfer has its limits too. ICL packages are mostly task-specific, so the gain weakens the farther you move from the source task. The RoboDojo Open group, which has no dedicated demonstrations, leans on a package built from another group; it benefits from extra interaction but has the lowest success rate of all groups. On RoboLab, which has no training set to begin with, it beats the listed baselines yet its absolute success rate only lands in the middle.
What, then, should future benchmarks measure? The report proposes three principles.
The third is especially grounded in reality, because the household prototype laid the limitation bare: the model forgets the refrigerator it left open, or its observations go stale while it deliberates.
ONTLE builds AI agents that take on the repetitive work piling up at small businesses that can't hire for it, with an "AI PM" at the center that drives each task to completion. Picture one scene. An AI PM takes on a small company's "month-end vendor settlement": gather tax invoices from 10 vendors, update the settlement sheet, and for any with outstanding receivables, draft a notice email and get the owner's approval. Nothing like a robot arm — yet this shares three things with the paper's long-horizon tasks. It steps through several subtasks in order (multi-step execution), it preserves intermediate state (state preservation), and it has to confirm whether each step actually finished (completion verification). The differences are clear too. Unlike a robot, a business system can re-query its records and undo a bad edit, so "spilled water" is less fatal.
The point where the design diverges is "completion verification." The model can report "settlement complete" while the cell values are wrong. The paper's residual failure #1 — "failure to verify physical success" — shows up in office work in exactly this shape. So the AI PM's settlement step, once the update is done, re-reads the settlement sheet and reconciles it against each vendor's ledger and the totals, and won't pass it as complete if they don't match (implemented). Progress state, too, is written to an external table rather than to the model's memory and re-read at every step, structurally blocking the "processes 7 of 10 vendors, then forgets the earlier receivable status on the 8th" failure (implemented). Whether performance drops when a new vendor comes in that the company's own operating manual doesn't cover — since that's only a limitation of ICL observed in robot benchmarks, not a result confirmed in our own settlement work — we hold as a hypothesis to validate against real data (a design proposal).
To put it plainly: general-purpose models have now reached "can solve it." Translating that competence into a product you can trust (verification), track to the end (state preservation), and afford (harness) is what AI PM design is for.
Map the settlement case onto your own product and check three things.
No. The report explicitly states that GPT-6-Astra's weights were frozen across every evaluation. The performance comes not from weight updates but from in-context learning (ICL) — feeding a knowledge package built from demonstrations into the context — and from the extended execution budget.