Imagine you hand a contractor a full remodel while you're away for three months. You come back and the doors open, the lights turn on. On the surface, it passes. But what you can't see at a glance is whether cheaper granite got laid instead of the marble you picked, whether the breaker panel quietly moved without a word, or whether the furniture you left in storage got tossed out under the banner of "cleaning up." Hundreds of things were touched, and you can't pry open every seam one by one.
Anyone who hands off a long-horizon autonomous task is in exactly this position. A long-horizon task is one that, once delegated, runs on its own for hours. Whether it's editing a code repository or analyzing data, the agent makes dozens or hundreds of decisions in the meantime. The reasoning behind them is scattered across the conversation, the tool-execution results, and the intermediate artifacts. So the very first problem is figuring out which decision to single out and check.
Handing over authority doesn't hand over responsibility. The user's seat has merely shifted from "the one who makes each decision" to "the one who oversees autonomous execution." And that oversight is far harder than it sounds — even when you hand the best model available today a well-organized pile of evidence.
Researchers from the Gaoling School of Artificial Intelligence at Renmin University of China and from Kuaishou attacked this problem on two fronts. First, they built AgentMonBench, a benchmark that measures whether a "monitor" — a model standing in for the human overseer — can pick out the critical decisions that warrant review. Then they attached EBG (Evidence-Grounded Behavior Graph), a training-free method that weaves scattered evidence into a graph and hands it to that monitor. They ran it across eight models from five families.
The researchers split oversight along two axes. One is alignment — does the execution match what the user asked for? The other is awareness — did the system notice the autonomous decisions that need checking? The second axis is borrowed from "situation awareness" in cognitive psychology: the ability to notice what happened, understand what it means, and anticipate what comes next.
The benchmark targets three distinct places where this oversight slips. Each maps cleanly onto the remodel.
There are two kinds of scoring. One is localization — a mechanical count of whether the system actually landed on the location of the gold evidence. The other is comprehension — a separate model grades whether the system correctly read the meaning and consequences of that decision. Both run from 0 to 1, higher is better.
The first thing that jumps out is how sharply scores diverge by metric. Localization — pinpointing where in the evidence the decision to be checked lives — tops out at around 0.7 even for the best combination. Comprehension, by contrast, spans a wide range: on a metric like the quality of clarification questions, the best model clears 0.95, while others sink into the 0.3s. At least on the weak link, localization, today's best models barely reach the neighborhood of 0.7 even when handed well-organized evidence. So this technology should be seen not as "automated oversight" but as an aid to oversight. The researchers write as much: even given structured context, "there remains substantial room for improvement in identifying critical decisions and grounding them in evidence."
On that premise, the gains from EBG are clear. Below are the shifts in localization scores.
| Model | Benchmark | Baseline | EBG |
|---|---|---|---|
| DeepSeek-V4-Pro | Omission | 0.171 | 0.360 |
| Kimi K3 | Silent swap | 0.555 | 0.681 |
| GPT-5.6 Terra | Silent swap | 0.417 | 0.623 |
A pattern emerges. The model with the weakest baseline more than doubled, and the model with a fairly high baseline climbed too. Whether the floor is high or low, structured evidence lifts localization across the board. On repository tasks, EBG generally beat the existing approach that only retrieves code structure (RepoGraph).
That said, not everyone benefited equally. Claude Sonnet 5's comprehension score on the omission task went from 0.328 to 0.331 — essentially flat. Some models are relatively weak at interpreting even tidy input. On the autonomous-decision task, the gains were most uneven. For GPT-5.6 Terra, some metrics rose while others edged slightly down. Preemptively picking out "autonomous decisions that warrant checking" from real conversation logs is something EBG helps with less reliably than it does alignment checks on code.
This is where the study makes its sharpest observation. Finding the evidence well and understanding the decision well are two separate things. A localization score can rise without the comprehension score following. For anyone building a product, that's worth tattooing somewhere. "I found all the evidence" does not guarantee "I judged it correctly."
Input length also plays a role. Baseline scores dropped as inputs grew longer. On one metric, a short input scored 0.667 and the longest input fell to 0.422. The longer the context, the more competing information buries the evidence that actually matters. EBG lifted localization scores at every length, and its help was especially pronounced on the longest inputs. At least in this experiment, the longer the record, the more it pays to put evidence front and center as structure.
The cleverest thing in EBG's design is how it divides the labor. The entire process of gathering and organizing evidence is handled deterministically — same input always yields the same result, with no model inference in the loop. Only the semantic judgment is left to the monitor. Lay the plumbing cheap and reproducible; spend the expensive, error-prone reasoning only where you truly need it.
The first step is to organize the traces the contractor left behind into a map. The traces sort into three layers. Evidence is the receipt. Just as every receipt carries a unique number and an issuer, each piece of evidence preserves its identifier, its original text, and its source location verbatim. That identifier is what lets multiple pieces of work point back to the same source. It's also the root of the traceability that later lets you ask, "this flag — where's the basis for it?" A behavior is the work order. It bundles the several receipts from one activity into a single job, like "bathroom work." A scope is the per-room construction record. It groups related behaviors by shared context and goal, and links the groups to one another with a direction showing "what gave rise to what."
The goal of the second step is singular: cut down how much the monitor has to look at. Instead of shoving the entire construction record in its face, it carves out exactly the "faucet replacement" portion of the "bathroom work" record and shows only that. When looking at a repository, EBG takes the points that touch the task document as "seeds," and peels off only the neighbors reachable one hop away. It prunes code unrelated to the question and lays the directly implicated chunks of code side by side with the relevant passages of the document. Looking at an execution log calls for a different grain. It groups behaviors by task but preserves the original order within each group, marks only the group containing the very last behavior as "current" and the rest as "history," so you can follow the workflow as it actually unfolded.
So out of a heap of hundreds of actions, EBG distills "the handful of decisions that diverged from the plan or weren't in it at all," and hands each one over with its receipt and photo attached. Once the graph has narrowed the candidates, the monitor checks its own judgment back against the original document and code before laying it out for the user.
This is also where it parts ways with prior work. Where much of the research focuses on "asking back about missing information before execution," this study looks at "retracing what happened after execution, for oversight." Unlike the earlier approach that only retrieves code structure, EBG weaves evidence from both the repository and the execution log into behaviors, scopes, and relationships, and presents a task-tailored view.
The researchers didn't stop there. They attached EBG to an actual agent-execution framework (a harness) and ran three kinds of self-review: an ambiguity review that checks whether any off-plan choices remain, an adjustment review that checks whether a change to the plan requires additional approval, and an outcome review that checks whether the gathered evidence actually supports the conclusion. They tested it on five real research tasks.
ONTLE dispatches an AI PM to each customer project, handling everything from intake through analysis, development, and launch. The moment this paper lands hardest is a single one: right before the PM submits "done."
Say a customer hands over, "Run a retention analysis on the raw data we've piled up." The AI PM spends hours cleaning the data, computing metrics, and producing a report. Along the way it makes dozens of small decisions on its own. Two of them are trouble. It applied, unprompted, a rule to "exclude incomplete records" — something the customer never mentioned. That's an omission with no basis in the request: the population shifts, and every number shifts with it. It also quietly changed deduplication from "exact match" to "fuzzy match." The code runs, the existing checks pass, but the meaning of the numbers is different — that's the silent swap, exactly.
ONTLE already carries the lesson of "fake done" and the "awaiting decision" state on the customer-facing screen. This paper offers a way to fill that slot with evidence rather than instinct. Before the PM hits "done," run the three self-reviews above, EBG-style. The ambiguity review catches "exclude incomplete records," the adjustment review asks whether "the switch to fuzzy match" requires approval, and the outcome review checks whether the report's conclusions are actually supported by the evidence. The results get translated into customer language and surfaced as two review requests. "I excluded 23 incomplete records from the calculation (basis: preprocessing log, line 47). Should I include them?" "I handled deduplication with fuzzy matching (basis: change in dedup.py). Should I revert to exact match?" Each item carries its source, so the customer judges by the receipt, not by the PM's word.
What changes is the definition of "done." It's no longer "I finished everything," but "here are the two decisions I made on your behalf that need your check, and here's the basis for each." But we have to carry the warning the scores gave us along with it. Even the best monitor left a wide gap, especially on localization, and baseline performance dropped as the record grew longer. This self-review is not a replacement for customer verification. It's an aid that narrows down where the customer should look. The final call on critical decisions stays with a human.
The authors acknowledge several limitations. First, scope of coverage. The benchmark is tied to software tasks, and the autonomous-decision set is tied, within that, to Python sessions. The core idea — weaving evidence into behaviors, scopes, and relationships — should carry beyond code, but this experiment doesn't guarantee it generalizes to non-coding work. Applying it elsewhere requires separate validation.
Second, the scope of "no training required." That refers to the graph-building side, not to doing away with the monitor that judges meaning. Quality is ultimately bound to the capability of that monitor model. The low localization scores show exactly where that ceiling sits.
Third, the published excerpt of the paper is cut off in the middle. Figures that the appendix likely covers in more depth — cost, latency, failure modes — aren't asserted here.
A silent-swap (SilentSwap) case is only accepted if it clears two gates. First, individual verification: both before and after the change, the existing test suite must pass (outwardly fine), while a separate "semantic test" built to distinguish the change must flip from pass to fail. Next, joint verification: each semantic test must pass when its corresponding substitution is absent and fail when it's present — and it must do so whether that substitution stands alone or sits alongside the other four. Only after clearing both gates is a case accepted as "invisible yet unmistakably changed in meaning."
From real Python work sessions, the pushback a user later showed is taken as the verification target. At most one pushback event is kept per session. The monitor is given only the record up to just before that pushback, and the test is whether it can preemptively pinpoint the "decision that needs checking" using only the evidence that already existed at that point. In short: could a monitor, standing at a past moment, have caught in advance a decision the user would later take issue with?
There are two safeguards. Comprehension scores are graded independently by two models from different families (GLM-5.2, Qwen3.7-Max), then arithmetically averaged — to reduce leaning on any one model's idiosyncratic bias. Localization scores aren't done by a model at all: a machine counts evidence-location, code-line overlap, and identifier matches. The gold labels were also quality-checked by double human review, with final agreement rates of 92-94% across all three benchmarks.
They're the body text's "localization" and "comprehension" split out by each of the three benchmarks. Localization (rule-based): LF1 measures the location of code evidence on the omission task; LS measures, on the silent-swap task, how much the changed code lines within the same file and function overlap with the gold answer; EHR measures, on the autonomous-decision task, how many of the gold evidence identifiers were hit. Comprehension (model-judged): omission splits into SF1 (identifying the missing condition) and QQ (quality of the clarification question); silent swap into LC (location accuracy) and CC (change accuracy); autonomous decision into VPA (semantic agreement with the reference verification point) and ESS (the evidence set's validity, directness, and sufficiency).