OpenGCUA, released by Alibaba's Token Hub (the MAI-UI team), is the first report from a project that aims to open everything needed to build a "computer-using AI" to the community. If you want the short version of what changes, there are two things.
First, when an agent operates a computer by writing code instead of clicking around the screen, task scores go up and the number of steps goes down. Second, when the model gets to make decisions that the harness (the scaffolding that wraps an agent and controls its execution) used to hardcode as rules, efficiency improves.
One caveat up front: there is no newly trained model in this report. These are results from running existing models on top of this design, and every model they tested improved. The gains came not from smarter models but from a different way of working.
Research on computer-use agents (CUAs, AIs that operate a computer on your behalf) has long been locked into a single picture: show the model a screenshot and let it drive the screen with a mouse and keyboard, imitating exactly how a person uses a computer. This is called a GUI-only agent.
This approach slows down as screen-hopping repeats. Take a task like gathering sales figures scattered across several Excel files and entering them into a different business app. A GUI-only agent opens one file, takes a screenshot, reads the values with its eyes, switches windows to the app and types them in, then goes back to the next file and repeats. As the number of files grows, the "look-decide-type" round trips grow right along with it, and each one requires calling the model again. Done in code, the same work often reduces to a single script that reads all the files at once, merges them, and feeds them into the app.
That is why the most advanced systems have started mixing GUI operations with shell commands (computer commands you type into a terminal), the accessibility tree (a channel that reads screen elements not as coordinates but as a structure of role, name, and value), and various APIs. When OpenAI shipped GPT-6 Astra, it also shifted its computer-use guidance toward "write GUI operations as code, bundle them with computation, and handle it all in a single output." The question has changed — from "how well can the model drive a GUI" to "how well does the system actually finish the work on a real computer, whatever means it uses."
The trouble is that these flagship implementations are almost entirely closed. For GPT-6 Astra, not just the model weights but the full prompt setup, the real-device harness, the device-control SDK, and the execution traces are all unavailable. So the community cannot reproduce the system, compare it against other designs, or examine how a given design choice shifts cost and success rate. OpenGCUA is a proposal to open up that closed part and study it together.
The authors do not define a good agent by how many parts it has. You can assemble every component — model, screen perception, task orchestration, tools, execution environment — and it still won't matter if you handle with ten screen clicks what a single shell command could do. So they define a "generalist computer-use agent (GCUA)" by three decisions made at every step.
Actions and observations pass through four channels: GUI (clicking the screen), the accessibility tree, the CLI (operating the computer through commands), and APIs. Each has its own strengths. The GUI works anywhere pixels are visible but is slow; the accessibility tree grabs elements directly without coordinates. The CLI is strong at handling files and data in bulk, and APIs reach functionality you can touch neither through the screen nor the command line. No single one is always best, so the key is to pick the right channel at each step and mix them when needed.
The first axis of decision is code execution. The model mixes screen operations like pyautogui.click(), shell commands run through bash(), and Python computation inside a single program.
The difference becomes clear when you compare it with the existing "batched action" approach. Batched actions have the model decide, in a single call, all the actions it will take and emit them at once. You have to fix the next click before you even see the result of the first one. Code execution decides the next action while the program is still running. Within a single call, you get conditionals ("if a dialog appears, close it"), loops ("fill three fields for each row"), and checks ("verify the status says Saved"). Twelve round trips collapse into one. The authors call their own earlier work on batched actions "programs without control flow," and judge it a stepping stone rather than a destination.
There are two benefits. Because intermediate results determine the next action, the model gets called again less often, and the programming ability the model already has can be put to work directly on operating the computer. In a case where the authors dissected GPT-6 Astra's actual execution traces, five recurring patterns emerged: cross-channel combinations where information from one channel is picked up and used by another; bundling 25 screens into a single contact sheet (one image that collects many screens in a single frame) to use as the next observation; checking background images and transparency with assert (code that tests whether a condition holds); and more.
The second axis of decision is handing to the model the choices the harness used to hold. A traditional harness hardcodes many decisions as rules, like "take a screenshot at every step." OpenGCUA lets the model make these decisions itself. In the current implementation, the model requests screenshots when it decides it needs them, decides how to track task state, and states a program's purpose before running it.
Why does this connect to cost? An AI processes input by turning it into tokens (the small units into which text and images are chopped for processing), and the more it processes, the more it costs. A cache is a mechanism that reuses the front part (the prefix) of the input without recomputing it when that part is identical to last time. Merely appending a new screenshot to the end of the input does not break the prefix you've already accumulated. What breaks the reuse is deleting or swapping out past screenshots to trim the context, which changes the prefix itself. The paper's implementation keeps taking a screenshot at every step but only chooses whether to feed it to the model. When the model decides "no need to look at the screen right now" and inserts fewer screenshots, the context grows slowly, past observations get deleted or swapped out less often, and the prefix survives — so the reuse holds and cost drops.
Lining up the existing families against the same frame (the three decisions) makes the differences visible. GUI agents fix all three decisions: they always look only at a screenshot, do one GUI action at a time, and follow a context rule like "the last few frames." CLI agents can't see the screen, but in exchange they can use loops, branches, and variables in the shell — flexible, but unable to handle apps that only work graphically. Multi-agent hybrids have a conductor role split the task and hand it to a GUI worker and a programmer, but channel selection happens once per subtask, and only at the conductor.
The authors' earlier work, Qwen-UI-Agent, puts GUI and CLI in one action space but bundles the actions into batches — a restricted special case of a GCUA. GCUA removes all those restrictions. Instead of a fixed sequence of actions, it writes programs that span all four channels; instead of receiving a screenshot at every step, it decides for itself what to observe; and instead of harness rules, the model manages the task state. The novelty of this work is not in inventing a new channel. It is in aligning scattered approaches along a single axis — "who makes the decision" — and pushing that decision toward the model.
The evaluation stage is the offline set of OSWorld-v2 (a benchmark that measures the ability to carry out computer tasks). The comparison set is eight advanced models, with GPT-6 Astra and Claude Opus 5 named explicitly. The authors implemented their GCUA design in the OpenGCUA harness, gave the existing models access to code execution and the handed-over decisions, and then ran them. Since this stage has no in-house trained model, what this experiment measures is not model capability but "which harness you put the same model on" — in other words, the effect of the design.
Efficiency doesn't end at the success rate alone. The authors also look at steps per task (one step = one model call), output tokens, reasoning tokens (included in output tokens but reported separately), and cost. Delegation is only worth it when it produces a result faster than doing it yourself. That said, these metrics only measure efficiency inside the benchmark; they can't capture every condition on a real user's computer. That gap is addressed in the real-device section below.
| Comparison | Feature enabled | Score change | Step / efficiency change |
|---|---|---|---|
| Full GCUA setup vs. batched GUI-only | code execution + handed-over decisions | mean partial score +19.6 | steps per task −53.0% |
| Code execution alone vs. GUI+CLI baseline | code execution only | mean partial score +6.1 | steps −14.8% |
| Qwen3.8-Max vs. hybrid CUA baseline | code execution | mean task score +5.2 | steps −17.0% |
| On-demand observation (two models) | handed-over decisions only | task score held | cache hit rate +10.5pp, +14.4pp |
The mean partial score reflects how much of a task was completed even when it wasn't finished end to end.
The first row of the table is the heart of it. The full setup, with both code execution and handed-over decisions turned on, raised the score sharply and cut steps by more than half against the weakest baseline, batched GUI-only. The improvement showed up across all eight models, and it closed more than half of the score gap between the strongest and weakest models. That narrowing suggests the weaker models may have benefited more from the design improvement.
The second and third rows isolate the contribution of code execution alone, but the two figures use different baselines. The 6.1 is an average against the GUI+CLI baseline, and the 5.2 is the result for the single model Qwen3.8-Max. The share you should expect from adding code execution to a system that already uses a shell is closer to these.
The fourth row, on handed-over decisions, is presented more cautiously. When only on-demand observation was turned on (the model requests screenshots only when it needs them), the cache hit rate rose on two models while the task score did not drop. It earned efficiency without sacrificing performance — and this is a limited observation, namely the cache hit rate on two specific models.
The real-device results are a different animal. Across four actual macOS tasks, the gap between models widened far more than on the benchmark. With the harness held fixed, Qwen3.8-Max used 19.4× the reasoning tokens and 5.7× the output tokens of GPT-6 Astra, and still completed fewer tasks. The case study traces the cause to model behavior: pouring excessive reasoning into simple operations, struggling to adapt to real-device conditions, and differing in code-execution ability. The point is that no matter how well you build the harness, a gap remains in the model itself — which is why OpenGCUA has put training its own model on the roadmap as the next step.
The limitations the authors state are clear. This release has no trained model. The real-device harness and the cross-device SDK are preliminary versions. The real-device evaluation covers only four macOS tasks — too small a sample to call a trend — and the authors themselves treat it as case studies, not a definitive benchmark.
There are further points worth pressing on. First, the full setup's 19.6-point gain is against the weak baseline of batched GUI-only. Compared against a system that already uses code execution, the pure gain is hard to pin down from this number alone; code execution alone's 6.1 points is closer to that question, but it too is limited to the GUI+CLI baseline. Second, code execution grants the model permission to run arbitrary shell commands and Python. The safety boundary for putting this on a user's real device is not the focus of this report, so whoever takes it to a product has to design that separately. Third, the "the model decides what to observe" design has a flip side. If, while the model skips an observation, a file save fails or the screen state changes and it misses that, it runs the next code on a false premise and the error snowballs. You have to weigh the gain of fewer calls against the loss of fewer chances to catch errors early — only then does the design's trade-off come into view.
ONTLE is a service that dispatches AI project managers. The job is to lighten the repetitive work of small businesses and solo founders who find it hard to hire. Consider the weekly operations report an AI PM produces every Monday. It tallies last week's sales from an inventory spreadsheet, pulls amounts from a vendor's PDF settlement statement, checks review ratings in the shopping-mall admin, and fills in a report template — and this is exactly the "hopping across many screens" we saw earlier. With code execution, you could handle the tally with a script and verify on the spot, with assert, that the settlement total matches the admin's sales figure, finishing in a single program what would otherwise take twelve window switches.
We haven't applied this design to ONTLE yet. But if we did, it's clear what has to remain. Stating a program's purpose before running it only becomes auditability when the materials used as input, the execution results, and any record where the cross-check came out wrong are all left in the log together. When a report is wrong, you have to be able to trace back to which judgment went off, and for the moment a cross-check breaks — like the settlement total not matching the admin's sales — you have to set the boundary in advance so the AI stops and hands off to a person. Which model you put under it as the engine has to be measured too. As the real-device results showed, even on the same harness, one model can spend more than 19× the reasoning tokens on simple operations depending on the model.
It's a preliminary tool meant to give the computer, the browser, and the phone a single programming interface. It binds every operation explicitly to a specific device, app, window, or tab; keeps accessibility and vision as separate means of observation and operation; and lets one program coordinate multiple devices within a single task. It's a design for handling work that spans several devices as one flow.
Reasoning tokens are included in output tokens, but reported separately they show how much of its resources a model pours into "thinking." Since what drove the cost gap on real devices was which model spent excessive reasoning on simple operations, you have to break out this share — invisible when blended into the output — to see why cost diverges even on the same harness.