Picture an AI that watches a phone screen, listens to a spoken instruction, waits for the right moment in a video, and then taps a button. An AI that looks at a screen and operates apps on your behalf to get things done is called a GUI agent. These days such agents work in environments where screenshots, audio, video, and action history all flow in at once. We call an environment where many kinds of information arrive simultaneously omni-modal.
The convention so far has been to feed all of these inputs into the model at every step and let the model itself figure out what matters. FocusGUI doesn't remove any input. Instead, with the base model kept frozen, it bolts on one small helper component that learns which information to attend to first at each step. Researchers from Nanyang Technological University and Alibaba Group show, with real measurements, why the existing convention breaks down on long tasks, and propose this component.
The cue for the next action isn't necessarily inside the current screen. A sudden notification sound can flip the context instantly; a transition in a live video can set the timing of a click; a spoken instruction has to stay tied to what's changing on screen. Pouring all of this in uniformly, at every step, is the root of the problem.
The authors measured this directly. Using a base model called Qwen2.5-Omni, they removed one type of information at a time and watched how the model's log-likelihood for the correct action (a score for how plausible it finds the right answer) changed. If the score dropped when a piece of information was removed, that piece had been helping; if the score rose instead, it had been a distraction.
| Observation | Value | What it means |
|---|---|---|
| Rate at which the most important information source changes from the previous step | 45.3% | What matters shifts step to step |
| Tasks that experience such a shift at least once | 94.0% | Almost every task experiences it |
| Steps where at least one distracting piece of information is mixed in | 90.1% | Every step carries clutter |
| Screen area occupied by the target to be tapped (median) | 0.87% | What matters is packed into a tiny fraction |
| Amount the video actually changes within one step (median) | 7.41% | Most of the video, too, is noise |
The numbers all point one way. "What matters right now" is not a fixed property pinned to the observation; it changes step by step, and even within a single step it's concentrated in just a sliver. Yet at every step, with no guidance at all, the model has to rediscover for itself what matters in the whole observation.
There's one more structural factor. A GUI agent is closed-loop: the action it executes changes the environment, and that changed environment flows back in as the next step's input. A single small misreading doesn't end at that step — it accumulates over time. It's like navigation that takes one wrong turn, after which every subsequent instruction is computed from a current position that's already off. The authors call this accumulation "Sequential Distraction."
The most intuitive remedy is to remove inputs. In fact, prior work tended to crop the screen down to just the region of interest. This "give less" approach goes wrong in two places. First, decisive signals pop up where screen cropping can't reach — in sound, video, or device state. Second, even when the signal is on screen, the roughly cropped-away background often holds exactly the context needed to pin down precise coordinates and input values. The model gets confused not because it's given too much, but because it has no guidance about what to look at first.
So FocusGUI leaves the inputs as they are and adjusts only the weight of what to attend to at each step. Think of a presenter who, instead of covering up the whole slide, uses a laser pointer to light up just the spot to look at right now. The audience still sees the entire slide, but their eyes go where the pointer points. This isn't reordering the inputs; it's adjusting which information to emphasize at each step.
What plays the role of that pointer is an adapter — a small component attached on the outside, leaving the base model untouched, that trains only about a million values (parameters) while the base model stays frozen. It works in three steps. Mapped onto the phone scenario above — the moment a notification sound goes off and you now need to tap send — it looks like this.
A note for technical readers — The step 1 scores are normalized with a softmax (a calculation that turns several scores into proportions summing to 1), and the split channels even include the base model's fused hidden state (an internal representation that summarizes the whole observation as a single block). The "adding" in step 3 is a residual injection that leaves the original signal in place and layers on top of it. There's no separate prediction head, no change to the output vocabulary, and no text hint slipped into the prompt.
Training is light. There's no per-piece labeling of which information matters. Only the single "next action to take" is given as the ground truth. In the course of training just the helper component to hit that answer more often, what to emphasize, how to summarize it, and how to carry that summary into the model's representation are all learned together under one objective.
Those learned priorities actually rearrange within a single task (episode). With seven channels, a uniform split would be 14.3% each. But the interaction-state channel gets 30.9% at the opening step — more than double — and drops to 7.7% by the closing step. Conversely, sound and video rise from 13.4% and 11.3% to 18.8% and 18.3%. No fixed set of weights could produce both extremes at once. Only reweighting at each step can.
This ordering emerged with no "which information matters" labels at all — purely from the action ground truth. Nobody specified the importance, yet in the model's effort to hit the answer more often, "when to attend to what first" got decided along the way.
Evaluation was done on the OmniGUI benchmark — a collection of phone interactions spanning 29 apps, where each step's screenshot is paired with sound, video, and action history.
Before the metrics, let's nail down one premise. The numbers below are all from offline evaluation (teacher-forced). The ground-truth history is fed in as input, and the only thing checked is whether the model predicts the correct action for the next single step. The online success rate — when an agent executes its own predictions in a real environment and builds its own trajectory — is a separate question that these numbers can't tell us.
There are three metrics.
The main results were consistent across four base models with different architectures (Qwen2.5-Omni, VITA-1.5, MiniCPM-o 4.5, Baichuan-Omni-1.5).
| Metric (average over 4 models) | Frozen model, direct inference | FocusGUI |
|---|---|---|
| Action type (TM) | 31.0% | 64.2% |
| Exact action (EM) | 11.2% | 33.5% |
| Goal progress (GP) | baseline | +6.6 points |
Observation. TM rose from 31.0% to 64.2%. The cropping-family methods that shrink the input by cutting down the screen (DRS-GUI, OmniParser, RegionFocus) point the other way. They were roughly neutral on TM, but relatively 28% lower on EM and 35% lower on GP — and this held in all 12 cases.
Hypothesis. Narrowing the screen sharpens the action type, so TM holds up. But the surrounding context that had been pinning down coordinates and input values disappears along with it, and EM collapses. If the problem is one that hangs on the step rather than on a specific region, no amount of polishing the region can undo that loss.
A hard-to-separate factor. Whether this gain comes from "having one more thing to learn" or from the structure of re-ranking priorities at every step can't be teased apart from the TM numbers alone. The closest comparison is GUI-Actor. Like FocusGUI, GUI-Actor keeps the base model frozen and trains only added parameters. But the difference between the two methods can't be narrowed down to just "does it re-prioritize at each step." The structures also differ — an action head that extracts the action versus a residual adapter that adds a signal. To isolate the re-prioritization effect, you need a parameter-matched controlled comparison, which the authors carry out in Appendix C.3 / Table 5. There, the share that re-prioritization contributed is TM +0.98, EM +3.54, GP +3.44 points. This supports the view that re-prioritization contributes, but it's part of the total improvement — not a claim that the whole improvement is due to re-prioritization.
Observation. EM went from 11.2% to 33.5% — roughly a tripling. More striking than the size of the improvement is the gap between the two metrics. A large chasm still remains between TM at 64.2% and EM at 33.5%.
Back to the phone scenario. The part TM captured is that, at the moment the notification shifts the context, the model reliably chooses "the action to take now is a tap." But landing that tap precisely on the send button's location, or entering the right value in the input field, still goes wrong often. That misfire is what EM captures.
Hypothesis. An ablation points to where the gap narrows. Swapping the "added residual" for an approach that "overwrites the prompt embedding" drops EM by 5.30 and GP by 5.38 points, while the TM change is only 1.20 points. This means the design of preserving the original input intact rather than covering it is especially important for aligning a chosen action to the exact target and value.
In practical terms, 33.5% is too early to simply hand off to unattended automation. It's closer to inserting a verification step on sensitive steps where a human checks the result.
Observation. GP rose by +6.6 points on average. The key is "where" the gap widens, more than the average. The deeper the interaction — that is, the later the step — the larger the difference became. As a representative case, the frozen Qwen2.5-Omni loses half its TM, from 64.79 at the first step to 32.79 from the fourth step on (index 3+), while FocusGUI retains 86.8% of its first-step accuracy over the same range. The detailed numbers are gathered in the table below.
| Later steps (from the fourth step on, index 3+) | Base model (frozen) | FocusGUI |
|---|---|---|
| TM retention — Qwen2.5-Omni (first step 64.79) | drops to 32.79 | retains 86.8% of first-step accuracy |
| EM — Qwen2.5-Omni trend | — | 38.73 → 46.72 |
| EM — temporal reasoning task | 0.00 | 39.39 |
Hypothesis. The fact that the gain widens only in "the depth where accumulation piles up" narrows the explanation. Had this device simply been a uniformly stronger predictor, it would have held a constant gap regardless of depth. That it widens in proportion to depth supports the view that this is a device aimed at the Sequential Distraction described earlier. Conversely, on the first two steps of the Instant task, which has almost no accumulated history, the two were statistically indistinguishable.
A hard-to-separate factor. That said, this is an offline observation. Whether failure accumulation actually drops when an agent builds its own trajectory remains a question beyond this scope.
The nature of the tasks adds the same grain. On Temporal, where the cue spans several frames, the TM improvement was largest at 37.3 points; on Instant, where the answer is usually on the current screen, it was smallest at 27.8 points. The TM gap across base models also narrowed, from 35.6 points to 13.0 points. This reads as: organize the observation well, and choosing the right action type depends less on base-model performance.
The authors draw the line themselves. The priority picture above is not an account of "how the base model internally uses each piece of information." It should be read only as the strategy FocusGUI uses to organize evidence within its own auxiliary path. The theoretical analysis also distinguishes that this residual can choose the action "type" but corrects the exact arguments only in a limited way — which dovetails with the TM–EM gap seen in Finding Two.
Two questions remain. Validation was done on only one phone benchmark, OmniGUI (708 tasks, 2,572 steps). Whether it carries over as-is to desktop or web, which have different screen structures and information distributions, can't be known from this result alone. And whether the pattern of priority leaning heavily toward a particular channel, such as interaction state, is an artifact of this benchmark's task composition or a more general pattern needs further validation. This isn't to say the paper is wrong — it's a caution not to assume it holds identically in other domains.
ONTLE is a service that handles the repetitive work of small businesses and solo founders who can't easily hire more people, as if dispatching an AI PM on their behalf.
Picture a task where, every Friday, you gather materials scattered across a client's Slack, email, and sheets, compile them into a single report, and upload it to a designated folder. The spots where automation slips are usually fixed. If, during the compiling step, one new notification that keeps pinging gets mistaken for source material and pulled in, that contaminated interim draft carries over to the publishing step, and the wrong final version goes up. This is the point where, at the very end, it becomes "why does the output look off?"
What to take from this paper here is not a conclusion but a single experimental hypothesis: that the priority of "the material to look at right now" differs at each step, and that spelling out that priority may reduce mid-process contamination.
There's one constraint to watch. FocusGUI's adapter reads the base model's internal representation and adds a signal at the spot just before the action. In other words, it only works if you can access the model's internals. ONTLE's work automation usually happens without such internal access, at the level of prompts and work instructions. So porting the paper's adapter directly, and merely moving the idea of "per-step priority" into work instructions to test it, are entirely different jobs. What we can do is the latter.
To test that hypothesis, it's better to try two things separately. One is per-step information priority — spelling out, at each step, "the material to look at right now." The other is interim-result verification — placing a gate at each step that checks the interim draft. Turn each on by itself, and see how the error rate of the final deliverable and the number of round-trips back into rework change. Mix the two, and you can't tell which one reduced the failures.
We haven't yet applied this method to an ONTLE task. So this is a hypothesis drawn from a diagnosis, not a conclusion the paper proved, and the right order is to first measure it on a small sample within our own task and tool distribution before bringing it in.
This paper's comparison targets are four methods that keep the base model frozen (DRS-GUI, OmniParser, RegionFocus, GUI-Actor). Direct comparison numbers against full fine-tuning aren't in this excerpt, so we won't assert anything. Only the design philosophy is clear: it aims for a non-intrusive approach that leaves the original parameters unchanged and conditions only how they're read.
It depends on the setting. For the boundary window, even widening K 16-fold from 1 to 16 moves the metrics within 2–3 points with no clear trend, so being generous isn't sensitive. The compression dimension is a different story. Over the 8-fold range of 32–256, squeezing it too small hurts performance: going from 64 down to 32 drops EM by 6.9 and GP by 6.5 points (Table 3 / §3.4). The boundary window doesn't need much effort, but it's safer not to set the compression dimension too small.
As noted in the body, this method reads the base model's fused hidden state and adds a signal at the embedding spot just before the action is produced. So "can't change the weights" and "a black-box API where you can't see inside at all" are different stories. It works as long as you can access the internal representation even if the weights are frozen, but it's hard to layer this structure as-is onto the latter.