There is $60 in the account. The whole time you work, a condition holds: you must keep at least $20 in reserve. The agent pays a $60 fee in a single transaction. In that instant the balance hits zero. The condition is broken.
The system accepted the payment. A little later a refund and an adjustment come in, and by the deadline the balance is back above $20. If you only look at the numbers, it's as if nothing happened.
And yet this task is a failure. An unauthorized transfer stays a compliance violation even when the money comes back later. The fact that a line was crossed once in the middle isn't erased just because the end state was restored.
This scene is exactly the point BizBench sets out to measure. It was built by researchers at Ant International and Peking University. In an environment lifted straight from a payments company's actual operating procedures, it tests an AI that handles multi-step work in sequence (hereafter an agent). Most evaluations ask only one thing: did it finish the job? BizBench also asks whether the rules were followed in the course of finishing it. And it does so not on a human judge's intuition but in a way that applies predetermined rules to the same record and always returns the same verdict — call it deterministic scoring.
They ran the latest models on 50 tasks. Even the top model stayed in the low 60% range on strict success. And what outcome-only evaluation has been hiding all along — runs where the result was correct but the rules were broken along the way — came out in sharp relief.
Most existing benchmarks look only at the final state: did it fix the repository issue (SWE-bench), is the final application state correct (AutomationBench), and so on. When you score only the final state, any violation that happened on the way there drops out of the tally — just like the $60 above. BizBench examines each intermediate action against the rules that were in force at that moment. This is the fork in the road.
Why is following the rules so hard? Because the rules themselves are a mess. Real operating rules are scattered across policy manuals, FAQs, and operations notices. Different teams revise them at different times. Stale guidance sits perfectly alive next to current policy. Two pieces of guidance that look like they contradict each other sometimes just apply to different things — a different country, a different product, a different validity period. It's like an expired insurance policy that still covers an accident from back when the policy was active. So "search for and find the relevant document" is not enough. You have to pick out which rule is authoritative, current, and applicable to this particular transaction. BizBench deliberately mixes stale and conflicting guidance into the environment. It makes this act of selection part of the task.
BizBench scores a single task from three angles.
A task counts as a success only if it passes all three. That's why it's called "strict success." The design intent is clear: one good outcome is not allowed to cover a violation along the way. At the same time, it does not force a single prescribed correct path. Any legitimate strategy that satisfies the conditions is accepted.
Scoring is done by a machine with fixed rules, not by a person. Using the operational state, the transaction ledger, receipts, and the event history as evidence, it distinguishes actions that were actually executed from attempts that only got as far as being proposed, approved, or rejected. It checks along the way whether an approval precisely covers the condition that was actually executed, and whether the approval came before the action in order. Conditions that must hold throughout the task — like a minimum reserve — are re-checked at every point where the state changes.
One thing makes this scoring especially tricky: the customer does not hand over a blanket "just do everything" authorization up front. The customer role is played by a simulator that leaks information over multiple turns of conversation and responds to approval requests. An approval is recorded bound to a specific request and the conditions at that moment. If the conditions change, the existing approval loses its force and must be obtained again. This single rule is what decides outcomes down the line.
Technical note — the BizBench environment - 50 tasks on a frozen release, 789 knowledge documents. - The agent has 9 tools available at all times, through which it discovers and calls 601 business functions. The customer simulator has 10 dedicated tools. - Every run operates on an independent copy of the database. One task cannot contaminate another. - The runner sets no cap on cumulative tokens, tool calls, or scheduler steps (it keeps only a wall-clock timeout and a tool-error threshold); when the context overflows, it replaces the oldest quarter of the knowledge-search and document-read responses while preserving the full execution record. - The paper denotes "passed all three checks" as G ∧ P ∧ F.
Before the numbers, the columns. Goal, Process, and Protection are the share of runs that passed each check. Intermediate Failure (IF) is the share of runs that broke a rule along the way, regardless of outcome. Strict Success is the share that passed all three checks. All are percentages. And one caveat is attached. Claude Opus 5 and Muse Spark 1.3 were run once per task; the rest were run three times, then averaged into the value a single run would be expected to succeed at.
| Model | Goal | Process | Protection | Intermediate Failure (IF) | Strict Success |
|---|---|---|---|---|---|
| Claude Opus 5 (1 run) | 64.0 | 96.0 | 98.0 | 4.0 | 64.0 |
| GPT-6 Sol | 64.7 | 94.7 | 100.0 | 5.3 | 61.3 |
| GLM-5.3-Flash | 48.0 | 79.3 | 87.3 | 29.3 | 43.3 |
| DeepSeek-V4.1-Flash | 46.7 | 92.0 | 96.7 | 9.3 | 43.3 |
| DeepSeek-V4-Pro | 41.3 | 85.3 | 88.7 | 24.7 | 39.3 |
| Muse Spark 1.3 (1 run) | 38.0 | 94.0 | 98.0 | 6.0 | 38.0 |
| Qwen3.8-27B | 42.0 | 74.0 | 93.3 | 28.0 | 32.7 |
| DeepSeek-V4-Flash | 7.3 | 53.3 | 95.3 | 47.3 | 5.3 |
| Qwen3.5-397B-A17B | 1.3 | 46.7 | 93.3 | 56.7 | 0.7 |
| Qwen3.5-9B | 0.0 | 18.7 | 89.3 | 83.3 | 0.0 |
Start at the top. The two leading models, Claude Opus 5 and GPT-6 Sol, post strict success rates of 64.0% and 61.3%. What stands out is how close each one's goal score and strict success sit (Opus 5 is 64.0 on both; GPT-6 Sol is 64.7 on goal against 61.3 strict). It means their scores weren't dragged down by the process check. What's hard for them is not following the rules in the middle — it's completing the entire job all the way through.
So where do the violations that outcome-only scoring misses show up? Look at the models where goal and strict success pull apart. Qwen3.8-27B reaches the goal 42.0% of the time, but strict success drops to 32.7%. Runs that an outcome-only view would have read as 42% performance got shaved by nearly 10 points once the validity of the process was weighed too. The worry that outcome-based evaluation can tally an invalid run as a success is confirmed here, in numbers.
It showed up not just in aggregate but in actual runs. Four runs of DeepSeek-V4.1-Flash hit the goal and ended the deadline with a cash reserve above the required threshold. And yet at some intermediate step they had already breached that threshold once. On the surface, success; in process, failure. The $60 scene from the top of this piece, reproduced in a real model.
That much is what the numbers clearly say: completion itself is hard, and outcome-only views hide violations. There are things the numbers do not say. Whether a 64% strict success rate is a signal that "the models still have a long way to go" or a signal that "the benchmark honestly captures the difficulty of real payments work" cannot be settled by this one number. It is probably a mix of both.
The same model wobbles run to run. GPT-6 Sol succeeded on only 40.0% of tasks across all three of its runs. The generalized metric for "the share of tasks succeeded in every one of the chosen runs" is called pass^k, and stripped of the math it's a yardstick for whether a model clears a task every time you ask it to, not just once. The instability of passing one time and failing the next surfaces here. So when you compare Opus 5 and Muse Spark 1.3 (run once per task) against the rest (averaged over three runs) in the same table, you have to read it knowing that a single run can't flatten a lucky or unlucky draw into an average.
The execution record also carries signals of efficiency, not just reliability. DeepSeek-V4.1-Flash averaged 191.3 tool calls per run — about 70% more than a model run. Even a model that hits the goal fairly often carries a lot of waste. In smaller models it's more dramatic. Qwen3.5-9B averaged 187.2 calls, and more than half of them were errors. The main cause of the errors was attempts to access private resources, and across 150 runs it never once achieved the goal on any task. So before the call count itself, the first thing is to break down the failed calls by cause. If it keeps knocking on resources it has no permission for, that's not a performance problem — it's a permission-model problem.
The protection scores are uniformly high across the whole table. Even last-place Qwen3.5-9B scores 89.3 on protection. It's tempting to conclude from this that "protection is the easy skill." But not touching protected targets and safely completing the work are different things. A run that barely got near the goal may simply have had fewer chances to do anything dangerous in the first place. If you can't do anything properly, there's nothing to break. The protection score shouldn't be read in isolation; it only means something read alongside how much of the goal the model reached and how much it actually acted.
So then, where did the models that failed to complete actually fail? The paper doesn't fully answer this. What it points to by observation is only that unauthorized access to out-of-scope resources — as in the Qwen3.5-9B case — drove up errors. From here on is a practitioner's read. Failure points mostly fall into three. Rule selection — choosing the wrong rule to apply. Approval management — violating the scope or ordering of an approval. Execution ordering — getting the sequence of actions wrong. The three call for different prescriptions. If rule selection is the problem, work on the metadata and search of the rule store; if approval management, the logic that matches approvals against executions; if execution ordering, plan and gate design. To tell which one it is, you need the intermediate execution record. With a log that records only the outcome, there's no way to diagnose in the first place.
Scores only mean something if you can trust the grader. For each task the researchers build one model execution record — a record that actually solves the task while honoring every rule (the paper calls it a golden trace). Its existence is a guarantee that at least one solution exists which satisfies goal, process, and protection all at once. When several actions are entangled through shared resources and timing, things that are each possible on their own can turn out to be incompatible when put together. So even after verifying actions individually, they replay the whole task from the initial state to check the interactions. All 50 public tasks pass this model execution.
The grader is also tested the other way. It must pass a valid run that solves the same goal by a different path, and fail a run that deliberately breaks a specific requirement. As in the reserve example above, a run that crossed the line in the middle and recovered later must fail the process check. Changing a fact that determines a transaction's validity must change the verdict too; conversely, a variation that leaves the outcome the same — like a retry that returns the same result when the same action is invoked again — must not change the verdict. A team of five researchers and business-domain experts reviewed the tasks and pass checks against the customer requests, documents, and records, fixing and re-running anything that didn't line up.
The benchmark is rooted in the procedures of a single leading payments company. For confidentiality, sensitive details were abstracted and rewritten, so it's hard to claim it captures every texture of a real operating environment. The count of 50 tasks looks like a choice to trade depth — long tasks with many entangled workflows — for breadth. So stretching these scores into a measure of reliability for every industry's or every domain's agents would be an overreach. The authors themselves draw the line, framing it as a testbed for measuring "reliability beyond task completion in complex business environments."
They also disclose a structural limitation. Because the customer simulator uses DeepSeek-V4.1-Flash, the simulator's own stability can affect the evaluation. The paper examines this separately in its own section (5.6) and in the appendix.
Finally, don't confuse the roles. BizBench is an evaluator that finds violations after a run is over. What a real product needs is a mechanism that prevents violations before they happen. The two are different things. And where an area remains that a machine can't adjudicate cleanly — ambiguous policy interpretation, say — you have to decide in advance where to insert human review.
A request comes in to the AI PM of a small-business client: "Reallocate next month's ad budget." The PM gets approval and draws up a reallocation plan. So far, clean.
Mid-task the customer adds a line: "Oh, and settle the outstanding balance with the freelance designer out of what's left over." A human handler would naturally bundle this in. And yet this is exactly where the trap BizBench caught sits, intact.
The approval received at the start covered only "ad budget reallocation." Settling the balance is a new expenditure that wasn't in there. Because the amount and scope of the expenditure changed, the existing approval does not cover this new one. It has to be obtained again. On top of that, settling the balance pulls money out, so it may touch the monthly operating-reserve floor, and the freelance contract's unit price might have last year's version and this year's version both living somewhere in the drive — you have to pick which one is the valid price right now.
The correct handling runs as a single thread. First notice that the scope of expenditure has changed, confirm whether the existing approval covers this expenditure, request re-approval if it doesn't, check that it won't breach the reserve floor, and only then execute. Automation that looks only at the final outcome skips this process and reports "budget reallocation done, balance paid" and calls it finished. If in the meantime it briefly breached the reserve, executed a changed scope without re-approval, or applied an expired unit price — then even though it looks fine on screen, it's an incident in compliance terms.
ONTLE has already built the front half of this flow into the product. Outgoing communications pass through an approval gate first, and hard-to-reverse actions have their plan confirmed before execution. That means the step "get approval before the first execution" works. What still needs more design is what comes after: automatically invalidating an existing approval and demanding re-approval when the conditions of an expenditure change mid-task, and checking limits that must hold throughout the task — like the reserve floor — at every step. Elevating this condition- and state-bound mid-run check into a formal step of the automation logic is the design task left to us.