AI agents that work in a terminal have a strange habit. Same model, same environment, and yet some days they solve the task and some days they don't. That's because an agent strings together dozens of slightly different commands over a single job, and each command changes the environment. The next decision is always made on top of that changed state.
What makes this genuinely painful is that some of those changes are hard to undo. Over-salt a soup and you have to throw the whole pot out. If the agent installs the wrong package or runs bad code, that command can corrupt the environment and get in the way of the steps that come after — though how recoverable it is depends on the command and the environment. The paper itself only says a bad command can hinder what follows. And when a move does turn out to be hard to reverse, it doesn't matter that the model knew a better one — by the time the bad command has gone out, it can be too late to take back.
People have tried to attack this before by spending more compute at inference time (test-time compute): draw more samples, verify the finished result, fix it, and run again. Most of these approaches correct things after execution. This paper pulls the question one step earlier. If you could filter before executing, under what conditions does the success rate actually go up?
The answer was a little surprising. Drawing lots of candidates mattered less than the quality of the "verifier" that picks the good one out of them. The bottleneck wasn't generation — it was selection.
The method the paper proposes is called Mid-Harness, and the design is right there in the name. It inserts a layer in the "middle" — between the model that produces commands (the generator) and the execution environment that actually runs them (the harness).
An ordinary agent runs a command the instant the model produces it. Mid-Harness instead draws several candidate commands from the same conversation history at that point. It then uses a verifier to pick one, and only the chosen one goes to the harness. The rest are discarded. At the next step, it draws several again from the now-changed history.
The nice part is how little it touches. Model weights, serving stack, execution environment — all left alone. It amounts to adding "draw several, return one" to the thin wrapper around the model call. That lets you change one thing at a time — the number of candidates, the comparison method, the verifier's strength — and cleanly isolate what actually drives the gains.
There's a catch with the verifier. It receives the task description, the history of observations so far, and the candidate commands — but not the reasoning the model did internally while producing each candidate. In other words, it has to judge "what this command would do in this environment" without running it. That's the fundamental difficulty of verification.
There are three ways to make that judgment. Take two commands as an example. ls -l shows you in detail what files live in the current folder; pwd just prints the path of the folder you're currently in. When a job gets stuck, whether you should first survey the situation (use ls -l to see what's there) or just confirm your location and move on (pwd) depends on the moment. Here are the three ways to pick between the two:
ls -l beats pwd." It makes the most calls.One intuition matters here. For the model too, choosing which of just two options is better — a relative judgment — is a much easier and more accurate task than ranking many options at once or assigning each an absolute score, because the comparison gets a shared frame of reference. But you pay for it. With 8 candidates, listwise needs 1 call, pointwise needs 8, and pairwise — comparing every pair — needs 28.
When the generator and verifier are the same model, this is called self-verification. You can also attach a different model as the verifier. As we'll see, that one choice made a large difference in the results.
The experiments run a base agent model on terminal tasks and compare it while varying only the verifier. Performance is read through two metrics. Each task is run 3 times: Pass@1 is the average success rate across those 3 runs (how well it does in one shot), and Pass@3 is the fraction of tasks solved by at least one of the three (how far it gets with a bit of luck).
First, they swapped in only the verifier — a very strong external frontier model. The result tilted clearly one way. A plain agent with no verification sat at 50.00% Pass@1; attaching a frontier verifier to the same generator and picking with even the simplest listwise method lifted it to between 64.63% and 68.03%. In other words, the generator was already producing the better command. The place it got stuck was the picking.
So wouldn't drawing more candidates fix it? No. In self-verification (verifying with the same model as the generator), doubling the number of listwise candidates moved Pass@1 from 49.32% to 51.02% — essentially flat. A weak verifier can't pick out the good one no matter how many candidates you add. Scaling the candidate count does not compensate for weak verification.
The key was not "how many you draw" but "how you pick." In that same self-verification setup, switching only the comparison method to pairwise reached 54.76% Pass@1 and 71.43% Pass@3. The verifier model stayed the same; just changing the way of picking to "pit them two at a time" raised it.
You can layer on one more step: distillation — copying the frontier verifier's judgments into your own model. The 117,000 pairwise responses produced by the strong model were injected not by retraining the whole model but via lightweight training that adds only a small set of extra weights (LoRA). These weights are turned on only for the verifier, not the generator. The result: pairwise Pass@1 rose to 57.14% and Pass@3 to 75.51%. Without touching the generator, they raised the success rate purely by making the verifier smarter.
What the distillation actually transferred shows up in an offline analysis that replays the saved trajectories. The scoring error (MAE) dropped from 2.59 to 1.05. The rate of picking the same side as the teacher model also rose: agreement on each individual pair went from 59.01% to 74.58%, and agreement on the final selection synthesized from those comparisons went from 38.52% to 57.79%. The strong verifier's "sense of what to pick" was transferred into a single small sheet of weights.
Good news comes with a cost. Pairwise makes many calls, so the compute spent on the model emitting tokens piles up heavily. The mitigation is "decision-only" verification, which skips the reasoning and outputs just the verdict of whether A or B is better. On the base model, this method held or improved performance versus the reasoning version while cutting token cost by 20.9% and 24.1%. On smaller and larger models, however, decision-only underperformed the reasoning version. That's a hint that long reasoning isn't strictly required for verification — but its benefit rides on model scale.
This method doesn't displace existing techniques; it stacks on top of them. A Best-of-N baseline — generate several finished runs and pick one — sat at 55.10% Pass@1; generating those three trajectories with Mid-Harness pushed it to 61.22%, and using the distilled verifier to 66.33%. That's 11.23 percentage points more from the same 3 environment runs. Combined with sequential refinement — summarizing the previous run to guide the next — Pass@1 reached 60.20%, with Pass@3 rising alongside. The authors report that a combination adding one round of sequential refinement on top surpassed Best-of-N at less than half the estimated cost.
The limits are clear too. Even distilled self-verification fell short of the frontier verifier. The nature of the remaining gap is interesting. The judgments where it diverged from the teacher clustered in the area of discerning "what the command would actually do in the environment" — 67.4% of the clear verification failures were here. Predicting a command's effect without running it remains the core challenge.
Another thing: verification weakens as the job runs longer. Agreement with the teacher fell from 68.13% in the early turns (turns 1–4) to 54.07% in the late turns (turns 17–32). That's a fact the paper observed. But let's not jump straight from there to "so humans should review more in the later turns." That's not an observation — it belongs to the domain of operational judgment.
Finally, all of this selection rests on the premise that the generator produces diverse candidates in the first place. If the generator only spits out near-identical commands, there's nothing for even the best verifier to pick from. And when the task count is small, the variance of a 3-run average is large, so small Pass@1 gaps like 54.42% versus 55.44% should be read with caution.
Let's move this into one concrete scene. A client company asks to integrate an external accounting program. ONTLE's AI PM has to create a collection table in this project's dedicated database and issue a command that stores the credentials needed to connect. If the first command the model comes up with gets executed immediately, one bad table design can tangle the entire downstream collection pipeline. The terminal agent's "hard-to-undo command" problem translates here into "an action that touches customer data and integrations."
First, one thing to be clear about. What this paper lifted was task success rate, not the safety of customer actions. Those are different goals. But the means share the same structure: a gate that filters candidates before executing. The pairwise comparison and distillation techniques that raised the success rate can be borrowed directly to make a safety filter — one that prevents fatal mistakes — cheaper and smarter. That's the practical use of this paper.
So in this scene the work has an order. First, pull out the hard-to-reverse actions: things with high failure cost, like table schema changes, writing credentials, and processing where money moves. Put a pairwise gate on only the most expensive of them — the table schema change. Just before executing, pit two candidates against each other and have it pick "the design that fits this collection requirement better right now." Turn it on selectively for only this list, not every action, to keep the cost down.
The conclusion "it's selective, so the cost is fine" comes with conditions. It holds only when such high-cost actions are rare across the whole, and when the verifier is accurate enough to be worth it. If your own model's self-verification is weak, first attach a frontier verifier to measure the performance ceiling. If a clear gain shows up there, internalize that judgment into your own model via distillation. This is exactly the path to moving verification ability over to our own model without leaning permanently on an external high-performance model — and it meshes with the model-routing standard ONTLE uses. If cost is still a burden, revisit the reasoning-skipping decision-only method alongside it — but since the benefit varied with model scale, compare it directly on the model you actually use before adopting it.
In short, what's more urgent for ONTLE is the safety side. This paper gives us the raw material for how to make that safety gate smart. Not a full rollout right now, but putting a gate on the single most expensive action to measure the ceiling — that's where to start.
No. If they're the same, it's self-verification; if different, it's an external verifier. The largest gain in the paper came from attaching a strong external model as the verifier. Distillation is the compromise: move that strong verifier's judgment into your own model so that in production you run a single model again while still getting comparable selection ability.
No. The roughly 117,000 comparison examples used for distillation are judgments (reasoning and preferences) the frontier verifier produced on its own. The teacher model plays the labeler's role, so no human annotation is needed — which makes the data that much easier to gather.
It would, but then the premise of "filtering before execution" collapses. Picking after running is already too late for commands that are hard to take back. The point of this method is judging without having run anything — and that very "predicting the effect without execution" is what remains the hardest part.
The paper's results were validated mainly on terminal tasks. The authors list extending beyond this scope as future work. That said, the skeleton itself — "select hard-to-reverse actions before executing" — carries over just as cleanly to other agents that call tools.