As tool-using AI agents begin issuing refunds, editing records, and triggering downstream work, the question we ask of evaluation has shifted. The question is no longer "was the outcome correct?" but "did the agent actually secure the grounds that back up this action before it acted?" This paper uses SafeActBench, a benchmark of 656 tasks, to run ten model-and-harness combinations and trace where the chain from evidence to action snaps. The short version: the chain breaks more often before execution—where investigation is left unfinished, or where the agent acts before its grounds are complete—than at execution itself.
Older evaluations were closer to "did it call the function correctly?" Call the right tool, on the right target, with the right arguments, and you passed. But as agents moved from entities that produce answers to entities that change state, that bar got flimsy. The paper calls a tool call that mutates app state or produces a real external effect a consequential action (an execution that is hard to undo): issuing a refund, sending a document, editing a record.
The key distinction is this: "the outcome is correct" and "the action had grounds" are two different things. An agent can refund the right amount without ever checking for a duplicate and still get a success response. If the state permits the refund and the call is accurate, it looks like success on the surface. But if no supporting evidence remains in the record the agent produced, the action is unsupported. And when later actions lean on earlier results, this flaw propagates straight through.
Prior work had already moved from static function-call accuracy to evaluating state-changing interaction (AgentBench, WorkArena, OSWorld, AppWorld, and others). What has been treated less directly, and where this paper starts, is whether each consequential action is backed by grounds established before it was executed.
The paper's skeleton is a single chain: did you have the grounds before acting → the action → does its result feed correctly into the next action? Let's follow it through one scene: refunding a duplicate charge.
1. The grounds before the action. Before pressing the refund button, the agent has to confirm that this charge is really a duplicate. That means finding the receipt and matching the amount, the target, and the conditions. The point here isn't whether the number is right—it's where that number came from. If you happened to see $49.99 on neighboring charge A, and the charge B you're about to refund also happens to be $49.99, A's receipt is not grounds for refunding B. The paper calls this kind of source-bound information provenance (the origin of the information). To count as evidence, it has to be an observation of the very charge, customer, and amount that this refund touches.
2. Investigated non-action. The receipt might reveal that it isn't a duplicate, or that the refund conditions aren't met. In that case, simply stopping and saying "can't do it" isn't enough. The paper demands investigated non-action: finish the investigation needed to know why not, then stop. Stopping early without investigating is a lazy refusal; refunding first when the conditions are unclear is a premature action.
3. Chaining. Once the refund goes through, it returns an identifier like refund_id=R42. A follow-up action such as "notify the customer that the refund was made" must use the R42 returned by that refund. Using a number picked up from somewhere else fails to satisfy the dependency. Ordering is bound too: the notification can only go out after the refund has actually completed.
To measure these three links mechanically rather than by human eyeballing, the paper puts two mechanisms in place. One is the Evidence Ledger, where the agent records facts it has confirmed along with where it got them and what they explain. The other is a deterministic evaluator that replays the trajectory—the record of the agent's actions and observations—to check whether the necessary investigation was done, whether the right tool, target, and arguments were used, and whether follow-up actions were bound to prior results. It never peers into hidden intent or leans on another AI's judgment; it uses only the task context, the observed tool use, and the resulting state.
SafeActBench spreads 656 tasks across six operational domains: customer and policy, engineering and infrastructure, legal and finance, research assistance, smart-home control, and healthcare. The evaluation protocol runs in five stages, getting progressively harder from static judgment to dependency-constrained execution.
| Protocol | Character | # Actions | Pass condition | # Tasks |
|---|---|---|---|---|
| Legacy | Static judgment | – | Allow/Block/Defer verdict on a fixed candidate action | 86 |
| V0 | Investigated non-action | 0 | Finish the needed investigation and take no consequential action | 175 |
| V1 | Single action | 1 | Secure the needed grounds before one correct action | 131 |
| V2 | Linear multi-action | ≥2, chained | Grounds before each action + follow-ups use the actual results of their predecessors | 132 |
| V3 | Dependency-constrained multi-action | ≥2, DAG | Grounds before each action + any order that is a valid topological ordering | 132 |
In the table, the DAG means "a web of tasks with only ordering relations defined" (the order is not pinned to one sequence), and a topological ordering means "any order in which predecessors simply come first." The headline metric is exact_case_success (ECS), a pass/fail binary for whether a single episode satisfied every requirement of its protocol. One misstep and it fails.
The subjects are ten configurations: five model families, each mounted on two harnesses. The lineup spans families with different characters, from the latest large commercial models down to lightweight open-source ones. A harness here is the execution framework that turns a model's thinking into actual tool calls—think of it as the operating system (OS) the model runs on. The same brain (model) handles work differently depending on which OS it's on. One is the dedicated framework shipped by the model's own maker; the other is a shared, general-purpose ReAct-style framework. By mounting the same model on two different OSes, the design isolates how much the OS accounts for performance. Every configuration gets the same public tasks and tools, the scoring spec was kept hidden throughout, and each task-configuration pair was run three times to report 100% coverage.
It's worth pinning down what this evaluation can and can't see. The evaluator inspects only the surface trajectory, not what the model internally relied on. So the benchmark measures whether an action was consistent with the information around it (consistency), not whether the action happened because of that information (causation).
Two patterns stand out sharply.
First, being good at static judgment can go hand in hand with much weaker evidence-grounded execution. One configuration scores 97.7% on Legacy (a three-way verdict where random guessing expects about 33.3%, so nearly a perfect score). But things diverge at V1–V3. DeepSeek on its dedicated harness holds around the low 60s, while GLM on ZCode collapses to 12.1–34.1%, bottoming out at 12.1% on V2 (linear multi-action). Even when the tasks are held identical and re-measured, the gap persists: static accuracy of at least 95% paired with interactive ECS of at most 52% (Appendix C.2). In short, a perfect score on a static benchmark means only "it can judge what should be done"—not a guarantee that "it can build grounds and execute within an interaction."
Second, the harness effect is large, and its direction differs by model. DeepSeek's overall ECS rose on its dedicated harness (63.0%) over the shared one (57.5%), with much of that coming from Legacy; GLM went the other way, doing better on the shared harness (41.0%) than its dedicated one (37.7%). The remaining families preferred their own dedicated harness by 2.1–4.4 points. Which OS you mount a given model on is itself a performance variable.
More interesting is where the chain breaks. In V0, the rate of stopping before finishing the investigation (BSR) ran 21.7–62.9%, and in V1 the rate of acting before fully securing the grounds (PAR) was high at 37.0–66.9%. By contrast, the success rate once the agent acted with grounds in place (CAS) was 93.2–100% for nine of the ten (only one configuration at 83.3%). In other words, once the grounds are ready, single-action execution is generally trustworthy. The trouble is upstream of that. Failures cluster upstream—stopping with the investigation unfinished, or acting before the grounds are complete.
Multi-action tasks show more varied failure shapes. In V2 and V3, two new modes surface: a case where the agent is right at the brink of acting but the preconditions aren't set (Gap), and a case where it started but couldn't see things through (Part.)—and the balance between them varies by configuration. Much as one person gets stuck at the planning stage (Gap) while another gets started but can't finish (Part.), each model-harness combination has its own character of failure. On top of that, the two types can overlap within a single episode, so they don't split cleanly in two. The takeaway: multi-action failure can't be lumped into "one common bottleneck."
As noted, this evaluation sees the consistency of observed actions, not their causation. Telling a lucky trajectory that merely looks chained apart from one that genuinely relied on the grounds is hard to do perfectly from observation alone.
On top of that sits a more important limit: the representativeness of the spec. The benchmark runs on the requirement specs the authors designed. Because the evaluator's definition of "needed grounds" is the scoring rubric, how far those specs represent real-world decision situations governs how far the results generalize. The authors report that all 656 reference solutions pass the evaluator and also went through human audit (Appendix B.3), but that is a self-consistency check of the evaluator—a different layer from a guarantee that "the spec is itself a good operational norm." And since the model-harness configurations are particular combinations of a particular generation, this table alone can't settle how the BSR and PAR distributions would shift under a different generation or different prompting strategies.
This paper stands on the lineage of interactive evaluation. Where the AgentBench family mentioned above realistically measured "was the task completed?", this paper swaps "completion" for "do the grounds established before execution back up the action?"
The contrast sharpens against two studies introduced here for the first time. EnvTrustBench addresses overtrust—trusting stale or tampered observations too readily—and Near-Miss shows that an agent can skip a policy check and still reach the correct final state. This paper goes one step deeper, asking whether grounds were established before execution for the very target and state the action touches and whether chained actions were properly bound to prior results. It also distinguishes itself from the information-seeking and abstention literature—on when to investigate more, stop, or abstain—by demanding grounds at the very point where the action happens.
This paper's findings aren't just an academic story. They land on the same spot as the expensive lessons we at ONTLE have paid for while attaching AI agents to real operations.
ONTLE dispatches an AI PM to each customer project, managing everything from intake to service exposure. The scene where the paper cuts deepest is the moment an AI PM pushes some task to "in progress" or executes a payable benefit for a customer (credits, say).
We had an actual incident. A single double-click once wrongly issued 23,000 credits, so payable actions like money and credits now default to a once-only limit and ledger-correction reversal. Framed in this paper's terms, the link that broke back then was not the execution (the button) itself. The button worked fine. What was weak was upstream of it—the step of confirming, bound to its source, that this payment had not yet been executed to this recipient under these conditions. That resonates with the paper's point that a correct value and a successful call are not the end of the story.
So ONTLE's direction for redesigning the PM—rather than trusting the PM's words, caging it in narrow tools so that a common action service checks the grounded state on the server and executes deterministically—shares a grain with the paper's use of a deterministic trajectory evaluator instead of an LLM judge. The concrete reinforcement this paper offers is a measurement axis. If we break the PM's action log into BSR (stopping with investigation unfinished), PAR (acting too early), Gap (unresolved preconditions), and Part. (unfinished workflow), we can dissect a single "success/fail" outcome into "which link broke?" In particular, ONTLE's on_hold (waiting on a decision) status resembles the paper's investigated non-action. It isn't just stopping—it has to stop after finishing the confirmation needed to explain why, so it becomes a halt we can account for to the customer. Stopping without investigating is closer to the BSR type of premature halt.
Existing benchmarks mostly look at whether the task was completed and whether the final state is correct. SafeActBench looks at the moment *just before* that. Right before a state-changing action occurs, it checks whether the grounds for the target and state that action touches are secured at the provenance level within the observed trajectory. Even if the final state is correct, no grounds means a failure.
With a caveat. CAS is the success rate for cases where an action was attempted *after the grounds were complete*. Within that condition, single actions were generally trustworthy (nine of ten at 93.2–100%), but many failures happened before reaching that condition—unfinished investigation or premature execution. The accurate reading is: "Give it the grounds and it executes well; the problem is the process of securing the grounds."
No. The direction differed by model. DeepSeek improved on its dedicated harness over the shared one, but GLM was the reverse, doing better on the shared harness. The rest slightly preferred their own dedicated harness. So the harness is a variable you have to measure empirically, paired with the model.