Could a coding AI take on the software work of a robotics engineer—say, for a robot arm on a car assembly line—instead of a human? On a new simulated exam, the answer is: only part of the way, and only on paper. Today's coding AIs are pretty good at writing code, running tests, and fixing what's broken. But when it comes to predicting how a structure they designed will hold up once it becomes a hunk of metal bearing a load, they're almost blind.
RLE-Bench, from researchers at Harvard and Georgia Tech, draws that boundary in numbers. Spelled out, it's a "qualifying exam for robot learning engineers." Pick the exam apart and one pattern keeps repeating: AI hits the targets that are scored clearly and explicitly. Hide the grading criteria, however, and it misses the outcomes that only surface later, once several conditions tangle together. This piece follows that one thread through three scenes.
Until now, most benchmarks for coding AI stayed inside the monitor. Patch a bug in a repo, grind through a long terminal task, train a machine learning model. Robotics benchmarks, on the other hand, do touch the physical world, but they usually only measured the performance of a single "policy." A policy is a robot's rule of behavior—"in this situation, move like this." It's like hiring a new engineer and grading them solely on "the success rate of the arm picking up a cup."
A field engineer's job is far broader than that. They assemble equipment, calibrate sensors, diagnose faults, write controllers within hardware limits, and retrain models when things don't work. RLE-Bench puts that entire process into a single exam hall. It fuses the old approach—measuring "coding ability" and "robot behavior" separately—into a new test of the coding ability that produces a working robot.
The exam's cleverest design choice is that it separates development from grading. The AI works its workspace within a fixed budget (time, interaction count, compute) and submits its deliverable. That deliverable is then handed to a grader-only environment for evaluation.
Three things used in grading are hidden from the AI: the random configuration values (seeds) that nudge the scenario each time, the privileged-state information only the grader knows (like an object's true position), and the answer-key files the grading is based on. Because grading happens on scenes and robot bodies the AI never saw during development, what shows through isn't a memorized answer but generalized skill. This "hiding" turns out to play a decisive role three separate times later on.
The 51 tasks are grouped into 9 families by shared character, and from there sorted into four kinds of work. Here's what each kind probes, with one representative task apiece.
The final score (the RLE Index) is the average across these four kinds with equal weight. The structure makes it impossible to climb the ranking by excelling in just one area.
The contestants on the exam stand The evaluation covers 11 "model-harness" combinations. Here, a harness is the toolkit and working environment that hands the AI model its "hands and feet"—reading and writing files, running terminal commands, and so on. If the model is the brain, the harness is the body. - The leading commercial models each ran on a dedicated harness (a coding-agent CLI) built by their own maker. - Open-source models without a dedicated harness were evaluated by borrowing the context-management strategy of a general-purpose coding harness. - The "reasoning effort," which makes a model think longer, was fixed to high across the board, and web search was blocked. The point was to measure whether a model reasons within the given environment, not whether it copies answers from search.
The leaders were two top-tier commercial models. What divided them wasn't coding skill but visual grounding—the ability to read what an object on screen is and where it sits. The gap opened widest on operating and inferring. Both demand reading and reacting to multiple sensory signals.
On traditional machine learning tasks like training, by contrast, the gap between models was noticeably narrow. That reads as a signal that the fundamentals of writing code and training models have converged among the top models. Still, it's too early to declare from this one data point that "coding and ML are solved problems." This is strictly an observation within this exam.
The same grain shows in the pose-estimation task. Most models scored higher on video that carries depth information (RGB-D) than on color-only video (RGB). But the real difference came from how they handled occlusion. When an object was hidden from view, the strong models didn't try to force a 3D position out of the depth data; they used that information for a simpler judgment—"is it occluded or not?" Notably, one top-tier agent trained a small neural network of its own to cover situations where depth couldn't be trusted, and it was the only agent that actually broke out a GPU on a task that left the method open. It means that even for simple pose estimation, once occlusion enters the picture, only the top models built a perception system worth trusting.
The evaluation feedback received during development clearly lifted performance. On policy-development tasks the AI can see a preliminary score mid-development, and in 29 of 36 sessions the final submission improved over that first score.
The problem is that the improvement stopped exactly at the line the feedback could see. On tasks where the eval set could be run ahead of time, mid-development performance landed within 5 points of the official score. But on robustness tasks that hid the test-time perturbations (the surprise variables shaken in) to the very end, the official score fell below the development score by 19 points on LIBERO and 41 on RoboTwin. A few AIs even guessed at the hidden perturbations and built their own mock exams, but none closed the gap.
Here a striking paradox appears: helper tools aren't always a help. One task's harness came in three tiers. L1 gives only basic commands (APIs—prebuilt bundles of functions usable on the robot), L2 adds a grasp-planning module that looks at an object and computes where to grip, and L3 even hands over answer information like object positions. Most models scored higher and cost less as support increased. But one of the strongest models was already strong on L1 alone, and adding support either didn't help or actually hurt. It's a signal that beyond a certain line, external modules can clash with a model's own judgment. That said, this pattern came from a single strongest model, so the evidence is still too thin to take as a general law.
The most painful scene came from designing.
The task of designing a mobile base to carry several robot arms was graded on structural validity, efficiency, stability, and task completion. Most models passed structural validity—meaning they did produce mechanically plausible designs. Yet those same designs frequently collapsed under physical evaluation, and all nine models scored zero on "static stability margin for the worst-case arm." The static stability margin is the headroom a robot has to stay standing without tipping over while at rest.
The reason, again, is the hiding. The brief spelled out every required property, but the verification protocol—what loads the grader applies and what motions it commands to decide pass or fail—was hidden. So the AIs filled in the visible design goals while entirely missing the outcomes that emerge from several forces tangling together, like instability under load or collisions during motion.
In a word, the AI sees the trees well (whether each individual part looks plausible) but can't see the forest (what happens to the whole once those parts trade loads, inertia, and contact). The small gap between models on mechanical design wasn't because they did well—it was because they all did badly together.
The authors acknowledge three limits themselves. Everything is measured inside simulation, so it guarantees nothing about real-world safety; the four-kind workflow captures only part of an engineer's job; and publishing the exam risks leaking it into future training data, which would undermine the evaluation's validity. So they recommend independent verification and human oversight before deployment.
I'd add two sharper questions. One: is the 41-point plunge on robustness really "because the model is weak," or is it an "unfair exam" that hid contrived variables even a real engineer couldn't anticipate? This data alone can't tell the two apart. Two: a single score that averages the four kinds equally flips the ranking in an organization where "control matters less and perception is everything." That's why you should look at the per-kind profile rather than the composite score.
Use this paper as a lens on our own work and the same "hidden laws of physics" show up as technical debt in software too. Let me follow one scene all the way through.
A customer asks us to "add a dashboard that integrates order data." The AI PM reads the spec and builds it. The screen renders, the numbers fill in, and "HTTP 200, DB row counts match" passes easily. This is the very picture of most models passing structural validity on the mobile-base task. On the surface, it's done.
The problem lies on the hidden side. Do the queries buckle under concurrent access at peak hours? Does the integration quietly fall back to empty values when the token expires? Do the totals drift because of exchange rates and time zones? These are all outcomes that surface late and indirectly. Just as in the paper, even when the brief lists every requirement, if the grading method is hidden the AI misses these tangled outcomes.
So what we changed is the definition of done. Done is not "the AI PM in charge of development passed its own tests"; it's passing a separate verification that runs under near-production conditions. When a developer trusts only their own test and reports done, you step right onto that same gap you saw on the robustness task—in production. At the same time, since feedback reliably builds skill within the range it measures, we load that separate verification with real failure modes (concurrency, token expiry, unit mismatch). Only once it passes the hidden test is it truly done.
A model's scores change the longer it thinks. If this setting varies by combination, you end up comparing "how long it was allowed to think" rather than the capability of the harness or the model itself. So the researchers set reasoning effort to "high" across the board. That said, this "high" is not the maximum—some of the evaluated models support even higher levels. And standardizing the setting's name doesn't mean the actual amount of reasoning spent by each model, or the harness's influence, was equally controlled.
Coding exams end inside the screen, and if you're wrong you can undo it. Robots are different. Results show up in physics, can't be undone, and forces like load, inertia, and contact tangle with one another. The ability to write each part plausibly and the ability to foresee the outcome when those parts intertwine are entirely different muscles.
No guarantees. Every measurement was done inside simulation, and the authors themselves nailed down that real-world performance and safety are a separate matter, calling for independent verification and human oversight before deployment. On top of that, the overall winner isn't guaranteed to be the winner on the axis that matters to your organization, so judge by the per-kind profile, not a single score.