Can you teach an AI to operate a piece of software without ever installing it? AutoGUIWorld says yes.
The trick is an unusual use of an image generator. Normally an image generator is for making pictures. This work repurposes it as a tool that draws how the screen changes when you take an action. Bolt a planner onto that — something to lay out the sequence of steps — and you can manufacture entire records of "look at the screen, click, type" practice without ever running the actual program. When the researchers fine-tuned an AI on this synthetic practice data, its scores on real desktop tasks and real scientific tasks climbed noticeably.
A GUI agent is an AI that automates software work the way a person does: it looks at the screen (a screenshot), moves the mouse, and taps the keyboard. But memorizing where the buttons are isn't enough. It has to develop a feel for things like "what does the screen look like after I press this button," "what stays and what disappears," and "how does an earlier step constrain a later one." That feel is built by watching a large volume of successful task recordings — trajectories that go from a screenshot, to an action, to the changed screenshot, and so on.
The problem is that these trajectories can only be collected from software that's actually running. The range of screen states and workflows an agent can encounter is entirely tied to whatever software happens to be installed. To widen coverage, you ultimately have to install new software, configure it, prepare the working environment, and keep it reproducible run after run. The more specialized the software — CAD, scientific computing, electronic design automation (EDA) — the more idiosyncratic its install dependencies and runtime requirements, with licensing and account constraints piled on top. The cost of building and maintaining each environment, one by one, is itself the real bottleneck for training. AutoGUIWorld's premise is to skip that cost entirely.
The clever part of this paper is that it splits data generation into two layers. It separates "what should happen" (the plan) from "so what should the screen look like" (the picture). To borrow a film analogy: a director with a script decides how the scene unfolds, and the camera crew shoots it frame by frame — except each frame is shot by picking up directly from the one just before it.
Let's follow a single task through the pipeline: changing the font color of a specific cell in a spreadsheet editor.
Working this way, the researchers produced 79,266 step-level training samples across four platforms (Ubuntu, Windows, macOS, Chrome).
The single most important concept here is "interaction fidelity." A plausible-looking screen and the screen a real system would actually produce are two different things. If you open a format dialog, the spreadsheet behind it must stay intact; if you changed the color of just cell B2, the other cells must be left untouched. The point is that the image generator can't simply crank out a nice-looking picture — it has to act as a model that changes state in response to actions, applying only the specified change while preserving the layout, style, background, and visible content. That's what makes the generated recording a coherent screen-action-screen chain rather than a pile of unrelated screenshots.
Underneath this approach is one smart act of giving up. The paper frames the situation with a model in which the agent never sees the system's true state in full and observes only a part of it — a partially observable model (POMDP). Put plainly, it's like driving in fog. The driver can't see the full blueprint of the road or the inner workings of the engine; they have to decide the next move from the lane and signs right in front of them. The computer is the same. The true state is everything — inside the OS, inside the apps, the file system, invisible internal values — yet the agent sees only one thing: the "screen." The core of this paper is that, rather than straining to clear the fog and recover the true state, it focuses purely on plausibly drawing how one "visible screen" turns into the next "visible screen." As a result, the recordings it makes can be chopped into individual steps and used as training data, and there's no need to keep the generator alongside the model during training.
The question you most want answered is this: does an agent actually get better at its job when trained on data made without ever running real software?
The researchers fine-tuned a large language model (training an already-built model further for a specific task) on the synthetic recordings and then ran it in real environments. The model they chose was Qwen3.5-35B-A3B. Scores went up on all four benchmarks, and they attached concrete numbers to two of them. On OSWorld, which measures cross-platform desktop tasks, the average task score rose from 33.0% to 40.8%; on ScienceBoard, which measures scientific workflows, the task success rate more than doubled, jumping from 14.0% to 32.2%.
Don't blur the two numbers together. OSWorld is an "average task score" and ScienceBoard is a "task success rate" — the metrics themselves are different. Combining them into some "average improvement of X points" would be wrong. The direction, though, is clear: practice data synthesized without any real software lifted measured performance on genuine desktop and scientific tasks.
That said, there's something these numbers can't prove. The synthesized screens capture only what's "visible" (window arrangement, content, style); they can't capture "invisible state" like file system contents, app internals, or a browser's internal structure. That's the price of a design that models only the visible screen. There's no real backend, so there's no way to check functional correctness — whether, say, "the click actually registered the invoice properly." This is exactly where it diverges from the approach of attaching a verifier (a device that checks whether the result is correct) to a runnable environment. That approach can verify correctness but has to pay the full cost of deploying and configuring environments, whereas AutoGUIWorld gives up some verifiable ground truth in exchange for broad coverage, cheaply. The two are less rivals than complements that fill in each other's gaps.
One more thing. This pipeline rests on the premise that "the planner's plan is correct." If the planner lays out the wrong order, the image generator will render even that wrong plan smoothly and plausibly. You end up with a recording that's wrong yet looks flawless. Quality control screens out poorly drawn scenes, but there's no clear guarantee it catches systematic errors in the plan itself. And the performance numbers come from a single specific model, so whether the same holds for models of other scales or families is hard to assert from this work alone.
Separately from performance, the researchers also measured, from several angles, how distributionally close the synthetic data is to real data. The short answer: pretty close. But close doesn't guarantee performance — which is why the performance numbers above and the distribution numbers below should be read separately.
The "real" data used as the baseline for comparison is six collections gathered from actual GUIs (ScaleCUA, AgentNet, aria_ui, WebSTAR, Mind2Web, OS-Atlas), with ScaleCUA — whose resolution is closest to the synthetic data — serving as the main point of comparison. Four yardsticks were used.
The weak spots show up in the numbers too. The one lagging most is Ubuntu. Comparing like-for-like within the same content regions, the discrimination test hit AUC 0.65 over wide, solid-color backgrounds — where the generation traces were most obvious. Conversely, text and icon fragments were almost indistinguishable on Ubuntu and Windows, at AUC 0.44–0.57. Small text and icons are indistinguishable from the real thing, while the generator's tell lingers over large backgrounds.
There's a scene ONTLE's AI PM runs into all the time at client sites. A customer asks to automate a repetitive task in their in-house ERP or a proprietary admin console. But no agent training data for operating that console exists anywhere in the world. And cloning the client's ERP wholesale to stand up a sandbox triggers licensing, personal-data, and operational risks all at once. This is precisely the "environment cost bottleneck" described earlier.
This research offers a new option for attacking that bottleneck. ONTLE could use the technique to build a cheap "digital twin" (a virtual replica modeled on the real thing) of a client console. Start from a handful of real console screenshots, extract that screen's theme, layout, and state as a seed, generate tasks that make sense on top of it, have the planner lay down the click-and-type sequence, let the image generator draw the next screen, and have grounding mark the coordinates to press. Pre-train an agent on the synthetic recordings gathered this way, and you can build up a good deal of the basics — "read the screen and know where to press" — before ever touching the real system. Access to the real system and fine-tuning on real data then become the step that closes the last remaining gap. Instead of standing up an expensive live environment in full, you cheaply build a "mock training ground" that lets the agent rehearse before deployment.
Of course, the line has to be drawn clearly. This mock training ground has no real backend. You can't verify inside it whether money actually moved in the customer's real DB, or whether a voucher was registered correctly. The visible screen isn't the system's ground truth. So this technique is strong for raising "perception and operation basics" ahead of time, but irreversible executions — payments, data changes — must not be left to it alone. Lift the basics quickly with synthetic data, but back operational execution with real-environment verification and idempotent design (a safeguard that makes pressing the same action multiple times yield the same result as pressing it once). In short, this research hands ONTLE a "cheap, fast mock training ground"; it does not remove the live-execution approval gate.
Within the scope of the public materials this article draws on, whether the data will be released is not confirmed. For the exact details, check the arXiv and HuggingFace pages in the references below.
The paper uses the image generation model gpt-image-2. It works image-to-image — taking the current screenshot as its base to draw the next screen — with the role of "a model that changes the screen's state in response to actions."
Because a single screen doesn't carry all the context — scroll position, a floating dialog, text being typed, dynamically changing page content, and so on. To pick the next action properly, you have to condition on the history of prior screens and actions as well.