Say you're teaching a robot to park a car. The catch: every training video shows nothing but a car parked cleanly. Now tell this robot, "This time, scrape the pillar." Nothing comes to mind. All it can picture is the car sitting there, perfectly parked. It has simply never seen a failure. A robot world model is exactly this driver. A world model takes a robot's action and renders a video of how the scene will change next — but if it grows up on nothing but successful demonstrations, it will paint a success even when you hand it an action where the gripper fumbles. DreamTrue is a study aimed squarely at this habit of "only imagining success."
A world model is a virtual environment for training and evaluating robots — in other words, a simulator. Instead of running a real robot, you roll the tape forward on "if the robot does this, here's what happens." For it to be useful, two things have to hold. It must follow the action it was given (action fidelity), and the way it shows objects being bumped and grasped has to be physically plausible (interaction plausibility).
The trouble erupts when you train such a model on public robot data. Two obstacles stand in the way.
First, the camera calibration is off. Camera calibration is the record of where and at what angle a camera captures the world. Models in this family draw the robot from its 3D model and overlay that onto the real video to signal "the arm is right here" — but this overlay only lines up if the calibration is accurate. Public datasets often lack accurate calibration, so the painted-in robot drifts away from the real robot in the footage.
Second, the data is almost all "success." It's packed with successful demonstrations, and failure footage is scarce. Here the very nature of machine learning trips things up. A model pulls countless examples toward the most common pattern. When success is overwhelmingly dominant, even an input that points at failure gets smeared toward "the most probable, average-looking picture" — that is, toward success. The gripper botches the grasp, yet the object still floats up after the hand as if everything went fine. The paper calls this habit "optimistic hallucination."
This is not a mere rendering glitch. When a simulator judges an action that would fail as a success, the entire evaluation of which control method is better gets inflated. A tool built to sift out the good options ends up clouding the judgment instead.
The textbook fix is obvious. Redo the calibration accurately, and collect fresh data that evenly covers failure cases too. But gathering new robot data costs enormous time and money. So DreamTrue changes course. Rather than collecting more, it repairs the data it has and fabricates the failures it lacks, widening the range it learns from.
DreamTrue solves this in three moves. First it corrects the misaligned parts of the data, then it makes the model imagine the failure cases it never saw, and finally it attaches a judge to score whether that imagination is physically plausible.
The starting point is fixing the broken calibration. It draws the robot from its 3D model and recorded joint values, overlays that on the real video, and then back-computes how far the two have drifted to correct the calibration. The striking part is the method. No dedicated capture rig, no depth sensor — it pulls this off with ordinary RGB video alone. Because it strips away the assumptions that earlier methods leaned on (a dedicated capture sequence or accurate depth measurement), it can be dropped onto most public datasets as-is. Calibration values recovered this way were released for three datasets, 153,666 episodes, 1,660 hours' worth.
It finds pixel correspondences between the rendered robot and the observed robot, between static backgrounds across frames, and between synchronized viewpoints (correspondences via RoMaV2, using SAM3 masks so only the robot region is matched). These correspondences are bound together under reprojection error and ray-coplanarity constraints, jointly optimizing the camera intrinsics and extrinsics, distortion, and even the relative offset of the arm-mounted camera all at once. Prior markerless methods needed a dedicated capture sequence; PointWorld-style approaches needed a sensor or accurate depth from FoundationStereo.
Next comes fabricating the failures that don't exist. A counterfactual is a situation that never actually happened but is posed as a "what if?" From a recorded action, it nudges the final point where the fingertip lands, so the grasp deliberately misses its target. Because the starting scene is left untouched and only the action is changed, the model gets to explore what a different choice from the same spot leads to. In parking terms, it's like fabricating and showing "a trajectory that cranks the wheel toward the pillar on purpose."
The last move is standing up a judge to score that imagination. A twisted action has no matching real future footage. That means there's no ground-truth answer to compare against. Yet physical plausibility is visible even without an answer key. When the grasp misses but the object floats up, a person knows it's wrong without any reference video. So they trained a "judge" model on data where humans hand-labeled defects in the robot, the object, and the interaction. This judge looks not at "does it seem plausible" but at "is there a defect." The released annotations cover 44,900 videos and 30,400 defect labels spread across the three axes (robot, object, interaction) — the 30,400 is a count of labels, not of distinct videos that contain a defect.
Now they put this judge to work as a trainer. Every time the generative model offers up a wrong-footed imagination, it gets penalized and retaught (reinforcement-learning post-training — a stage that further trains an already-trained model toward a specific objective). The reward is "the defect probability with a negative sign," so fewer defects means a higher score. They also wired in one safeguard. Recorded actions do have real future footage, so how closely the output resembles that ground truth is folded into the score too, keeping image quality from collapsing.
Counterfactuals are made by applying an SE(3) transform (a position/rotation perturbation) to the last end-effector (fingertip) pose of a recorded action, interpolating from the fixed initial pose to this perturbed endpoint, and then converting that back into a real joint-action sequence via inverse kinematics (IK). Post-training scores with a frozen reward model; the optimization normalizes rewards within a group GDPO-style before combining them, and updates the generator with DiffusionNFT. The reward model is a vision-language model (Qwen family) fine-tuned with cross-entropy to predict defect probabilities along three axes — robot, object, and interaction. Architecturally, the rendered-robot RGB and depth are encoded with a video VAE, while masks and camera rays (Plücker maps) are encoded with a 3D-convolution geometry encoder; the two are fused and injected into a video DiT through a VACE branch. Gripper openness is mapped to background RGB brightness, and multi-view prediction stitches the latent videos of several viewpoints width-wise to predict them in one pass. The training objective is flow-matching velocity.
The effect of the three mechanisms shows up not in any single number but in the relationships between the numbers.
Start with the headline metric. On the twisted counterfactual actions, the interaction-defect rate fell from 48.12% to 6.25%. Here, the 48.12% is not an outside competitor but DreamTrue's own early version (Stage I, the state before post-training). In other words, it's measured against its own starting line. The object-defect rate likewise dropped from 31.88% to 3.12%. Meanwhile the metric for how well it followed the action (nDTW) actually ticked up slightly, from 0.8735 to 0.8831. Usually, tightening down on defects loosens action fidelity — they dodged that trade-off. It wasn't entirely free: the overall image-quality metric slipped ever so slightly, but what they gave up is far smaller than what they gained.
What this change actually means is clearest in the policy-evaluation experiment. They fed 240 actions produced by four control methods (policies) into each world model, had it predict the outcome, and then had humans judge success or failure to compare against the real simulation results. DreamTrue's success-rate predictions sat almost right on top of the true values. The baseline models, by contrast, inflated the rate in the low-success regime — imagining failed grasps as successes. Back to parking: the simulator can finally picture the pillar getting scraped. As a tool for choosing between policies, it got that much more honest.
The training data combines four public datasets (AgiBotWorld-Beta, DROID, RoboMIND 2.0, RoboTwin 2.0) into 2,232 hours of real and simulated multi-view trajectories across five robot-arm types. For AgiBot, they separately built 160 counterfactual test conditions (54 tasks) that perturb recorded trajectories. The comparisons are DreamDojo, GE-Sim 2.0, Genie Envisioner, EnerVerse-AC, plus a version using only its own Stage I (w/o RL) and the final model (full).
There are four metrics. Action fidelity is nDTW (normalized dynamic time warping — how well the predicted trajectory tracks the target trajectory); visual fidelity is PSNR/SSIM/LPIPS (how closely it resembles the ground-truth video); overall quality is EWMScore-P; and physical plausibility is the majority-vote defect rate from three evaluators (agreement measured by Fleiss' kappa).
Supporting numbers for policy evaluation: the success-rate prediction error (MAE) dropped from 12.92 to 7.08 percentage points, the aggregate bias was +0.42 percentage points (nearly zero), and the rank correlation was strong. On the AgiBot benchmark the final model led all comparisons on both action fidelity (recorded 0.8772, counterfactual 0.8831) and visual fidelity (PSNR 23.06, SSIM 0.936, LPIPS 0.094), and it also placed first in the AgiBot World Challenge 2026 world-model track on overall score (EWMScore 0.829), action fidelity (0.9651), and visual quality (0.6246). That said, overall image quality (EWMScore-P) slipped slightly from 72.84 to 72.51, and scene consistency (0.8974) trailed the top teams (Loop 0.9024, PAIWorld 0.9041).
The trend holds when extended to other datasets. On DROID, versus Ctrl-World, PSNR 22.00→22.95 and LPIPS 0.162→0.073. Isolating the effect of calibration alone, using the calibration values in training and inference moved AgiBot nDTW 0.8159→0.8711 (synchronized positional error 7.73px→4.60px) and DROID nDTW 0.7718→0.8919 (error 11.78px→8.06px).
First, transfer to unseen robots and scenes still rests only on picture-level evidence. They show qualitative results — "it produces plausible predictions" on a held-out robot and new real scenes — but there's no numerical comparison. The quantitative basis for a strong generalization claim is thin.
Second, reward post-training comes with a trade-off. Push hard on the single objective of defect rate and other quality gets squeezed (the slight drop in image quality we saw earlier is the price). The risk of reward hacking — the generator converging toward fooling the judge — is also ever-present in reward-based training.
Third, the judge is tethered to human labels. Defect labels across some 40,000-plus videos are a product of cost and subjectivity. They refined them with three evaluators and agreement management, but the definition and boundaries of a "defect" can shift by domain and by evaluator.
Fourth, the counterfactual takes a single form. Right now it's just one method — twisting the fingertip pose. The effect is clear, but whether it covers failures that twist contact and dynamics in more varied ways is an open question.
Fifth, at the end of the day it's a video predictor. Its gains as a more honest simulator are real, but these gains alone don't close the gap to controlling a real robot (sim-to-real).
DreamTrue's novelty lies not in any single technique but in threading three moves into one. It repairs the data it has, fabricates the data it lacks to widen coverage, and scores interactions without an answer key. Action-conditioned world models already existed, and so did reward-based post-training, but the design that weaves all three into one pipeline is where it parts ways with other work.
Among action-conditioned world models, Ctrl-World tackled multi-view long-horizon prediction, while Genie Envisioner and GE-Sim 2.0 handled simulation for policy learning and closed-loop evaluation. In the push to broaden interaction coverage, DreamDojo drew knowledge from human video, PlayWorld from a robot's autonomous interaction, and FACT from failure trajectories. On the geometry and calibration side, BridgeV2W added robot state as pixel-aligned masks and EnerVerse-AC added camera-ray encoding, while classic hand-eye and markerless methods needed dedicated capture and PointWorld needed accurate depth. On reward-based post-training, WorldArena and EWMBench handled evaluation, and WorldCompass did post-training with rewards for camera-trajectory tracking and per-frame image quality. Where those rewards mostly looked at motion and generic image quality, DreamTrue shifted the reward's focus to physical object interaction.
For ONTLE's AI PM, the most painful spot is the "done report" junction. A customer request is broken into tasks, and a worker decides "is it okay to mark this task done?" The recurring failure here is the fake completion. An external system was hooked up but in reality not a single row came in, or the deliverable it produced diverges from the request — and the agent writes "done" anyway.
There's exactly one place where this overlaps with the robot world model. Both judge a failed job as a success. Just because the robot's optimistic hallucination came from training skewed toward success data, I'm not about to declare ONTLE's fake completions have the same cause. One is the statistical pull of a video-generation model; the other is a hole in a procedure that passes deliverables through without checking them. The inner workings differ. What matters is that the symptom — insisting a failure is a success — is the same, and the way DreamTrue handled that symptom maps directly onto redesigning ONTLE's completion gate.
Say a customer asks us to connect their accounting system and pull in their sales data. The PM breaks this request into "authenticate → collect → load" tasks. Lay DreamTrue's prescription over this and the completion gate changes like so.
First, don't raise the PM on success cases alone. Instead of showing only successful integrations, deliberately feed it failure scenarios. The auth key is wrong. The connection succeeds but only 0 rows come in. A file is created but it's empty inside. Using the same trick by which DreamTrue twisted fingertip trajectories to force failure into view, we twist the conditions of past completed tasks to synthesize "near-failure situations" and use them as a check set.
Next, separate the PM that does the work from the judge that rules on completion. The judge treats three things as defects: request misreads (work that diverges from intent), deliverable mismatches (a gap between request and result), and false completion reports (a "done" not backed by the DB or the deliverable). We raise this judge by hand-labeling real fake-completion cases that actually blew up in the past. Just as the robot judge looked not at "does it seem plausible" but at "is there a defect," this judge looks not at "does it look done" but at "is there evidence of done."
The last move is the most hands-on part for ONTLE. Just as DreamTrue didn't leave everything to the judge where ground-truth footage existed, but checked against actual image quality, ONTLE actually checks what can be checked. For this accounting integration, before asking the judge's score, we directly count the rows in the sales table that was just loaded. Zero rows and it ends right there. If the external API response code points to an auth failure, it ends there too. No matter how plausible the judge finds it, it returns not to "done" but to "awaiting a decision." Only in areas with no answer key — judgments like "does this deliverable match the customer's intent" — do we lean on the judge's score.
A completion gate reshaped this way never presumes success up front. A judge schooled on past failures dissects the deliverable, counts what can be counted and checks it against reality, and blocks completion when there's no evidence. It's the work of stripping the very optimism — the kind by which the baseline models inflated past the truth in policy evaluation — out of a product's verdict on "done."
nDTW measures whether the trajectory was tracked well; PSNR measures whether the frame resembles the ground-truth video. Neither captures physical plausibility — like "the grasp missed, so does the object float up?" A prediction can have a correct trajectory and a plausible-looking frame while being physically wrong. To fill that blind spot they set up a human-judged defect rate as a separate axis, and as it happens, that's the axis where the biggest improvement appeared.
It's a coincidence. The 48.12% in the summary is the interaction-defect rate of DreamTrue's own early version (Stage I). The comparison model EnerVerse-AC's interaction-defect rate is also listed as 48.12%, but the two are separate numbers pointing at different things.
The paper states it released calibration for three datasets (153,666 episodes, over 1,660 hours) and annotations for 44,900 videos (30,400 defect labels across the three axes — robot, object, interaction — not 30,400 distinct defective videos). If your work touches robot-video generation or evaluation, the reuse value is high. Check the specific access method and license in the arXiv paper and project page below.