An agent is never just the model. Everything around it — how you prompt that model, which tools you hand it, how you monitor and correct its runs — is what we call the harness. When it comes to improving this harness automatically, attention has so far gone almost entirely to the how: how do we fix it? The what to learn from — which failure cases we collect feedback on — has simply followed whatever order was fixed at the start. ActiveSaddler goes straight at that assumption. Change the harness and you change which cases are worth learning from, so evolve the training plan (the curriculum) right alongside the harness. Against the same optimizer with the training order frozen upfront, it improved test performance on both benchmarks.
A good harness can dramatically improve the stability of long-horizon, multi-step tasks, even with the same underlying model. The catch is that a harness goes stale the moment the model, the task, or the environment changes. That's the motivation behind Automated Harness Optimization (AHO): run training scenarios, diagnose defects from the logs, and automatically patch the prompts, tool interfaces, and control logic.
Think of a sports coach. The coach keeps refining how to correct form (the way the harness gets fixed), but sticks to the schedule drawn up at the start of the season for which drills to run today. So the athlete burns time repeating the same drill on a weakness they've already overcome, while a weakness that only surfaced mid-training isn't on the schedule and never gets touched. Existing harness optimization looks exactly like this: sophisticated about how to fix, but shackled on what to learn from — bound to a full training set chosen before anything starts, or to a pre-baked order.
In practice, under the fixed-order approach, a sizable share of the failures observed during training are still unfixed in the final harness (paper, Figure 3(b)). A fixed training plan misses two things. First, it keeps hammering failures it has already fixed, or walks away from failures it hasn't fixed yet too early. Second, the trade-off between revisiting a known failure and running an unseen case keeps shifting as optimization proceeds, and a fixed plan can't tune that balance. The tighter the rollout budget (the budget spent on a single run), the more this waste hurts.
Fix the harness and what's worth learning from changes too — that simple fact is where ActiveSaddler starts. Let's follow a single failure from end to end to see how it works.
Discovery. When an agent fails at a task, it leaves a trail behind (a failure trajectory: a log of where and how it got stuck). Say the same failure — "it keeps calling tools in the wrong order" — recurs across many tasks.
Grouping. ActiveSaddler takes these scattered, similar failures and bundles them into a single fixable defect. It calls this bundle a failure-pattern arm. Split too finely by task and the same defect fractures; lump everything together as "tool use in general" and distinct defects blur into one. The knack is to catch the defect at a resolution somewhere in between (how coarsely you group failures).
Estimating headroom. Next comes prioritization. Using "how much more this pattern could improve" (learning-progress potential) as the signal, it goes after the patterns with the most room to gain first. Because the same failure is worth a different amount of a second look once the harness has changed, this estimate is redone on every iteration.
Re-learn or explore. If there's still headroom, it picks the pattern back up and works it further (review). If one pass has mostly resolved it, rather than pouring more budget into the same pattern, it runs fresh, not-yet-tried cases to mine the next weakness (exploration). When many failures remain, review pays off; when few remain, exploration does — so the ratio between the two keeps shifting with the situation.
Each time these four steps complete a loop, the result feeds back. Every pattern's priority is updated, and the set of discovered patterns itself grows or shifts. That's the concrete meaning of "the curriculum evolves alongside the harness." Mathematically, the setup is modeled as a "pick the slot machine whose payout odds drift over time" problem (technically, a non-stationary bandit). The one twist versus an ordinary slot machine: the arms aren't fixed in advance — they're minted on the fly from failures observed during the run.
The clever part is that the mechanism for fixing the harness isn't touched at all. ActiveSaddler layers only a curriculum of what to learn from on top of an existing optimizer (the paper builds on the AutoSaddler setup), so it's more like a module you swap into an existing optimizer.
One underlying difficulty is worth naming. In harness optimization, both the curriculum's goal and its progress are invisible. A single scenario doesn't pin down a specific weakness; a weakness only shows up once you run it; and the same scenario produces different failures under different harnesses. On top of that, a diagnosed weakness has no direct signal — no loss value — telling you "still unsolved." ActiveSaddler groups failures into patterns and re-estimates their value over and over precisely to cope with this invisibility.
The key comparison is simple. Same optimizer, same rollout budget. The only thing changed is the curriculum. Against freezing the scenario order before optimization begins, ActiveSaddler lifted test Pass@1 (the fraction of tasks solved on a single attempt) by 4.4 percentage points on GAIA2 and 7.5 percentage points on Terminal-Bench 2.0 (a percentage point, or pp, is the absolute difference in pass rate).
The numbers look small, but on hard tasks that take many tool calls rather than one shot, a few points of pass rate is anything but small. What this says is that how you train the same model can move pass rate meaningfully — and it does so without touching a single weight.
There's an efficiency angle too. Figure 1 in the paper plots how the best-so-far dev-set accuracy rises as cumulative rollouts grow, and the shape is one of reaching a better harness faster within the same budget.
What drives the gain can be teased apart with ablations (removing one component at a time to measure its contribution). (1) Assembling the optimization target dynamically as you go, (2) estimating that target's shifting value, and (3) balancing review against exploration of fresh cases — drop any one of the three and the gain shrinks. This is where the paper's larger claim comes from: that curriculum learning deserves to stand as its own independent "axis of optimization" in harness optimization.
First, the evaluation setting. The experiments are offline, use the standard train/dev/test split, and are limited to just two benchmarks (GAIA2 and Terminal-Bench 2.0). Whether it carries over to other domains or to online, test-time settings is outside this paper. The size of the gain also varies by benchmark (4.4 pp versus 7.5 pp), so which task distributions see large versus small gains needs more study.
Next, the method itself. ActiveSaddler isn't a standalone optimizer — it's a curriculum layer you put on top. It only runs with a harness optimizer underneath (the paper uses AutoSaddler, and the appendix applies it to another optimizer, GEPA), and the reported gains are strictly relative to using a fixed order with that same optimizer. The hidden variable that governs performance is the resolution at which failures are grouped into patterns — too coarse or too fine both hurt — so curating the list of discovered patterns (the pattern registry) remains a quiet hard problem. The extra work of pattern extraction, value re-estimation, and exploration decisions carries cost, of course, and the paper devotes a dedicated section to cost analysis.
Finally, this is a technique that automatically rewrites an agent's control logic. That's also why the appendix separately addresses responsible-deployment concerns like safety, security, privacy, and governance. When you push this to production, checks on this axis can't be skipped.
Recall the "say-do gap" our AI PM used to have at ONTLE — the mismatch where it promises to send a deliverable but then skips the actual send action. ONTLE has already narrowed this considerably with a shared action service and server-side deterministic execution. This PM, too, ultimately runs on a harness: the prompt that interprets requests, the action tools it calls, and the control logic for when to stop and wait for a human decision.
Suppose we improve this harness using past request logs. The common approach is to "replay the last N requests in a fixed order, tweaking the prompt and control logic as we go." Put on ActiveSaddler's lens and the order changes. First, from scattered failure logs, bundle the "skips the promised send" mismatch into a single pattern. If one pass mostly resolves this pattern, don't pour more budget into the same spot — move on to a new type of request we haven't tried yet and mine the next weakness. Conversely, if there's still room to fix, stay on that pattern.
The crux of what changes is that no human pins down the list of what to fix first in advance. Each time the harness is fixed, what to learn from next is decided fresh by looking at that result. For a product like ONTLE that keeps updating its PM harness, this points toward automating the operational judgment of "which failure log do we revisit first." It isn't free, of course. The paper is an offline research setting with an optimizer and an evaluation harness already in place. To actually use it, you first need a consistent standard for scoring request success and failure, and the machinery to safely auto-patch and roll back the harness.
It never touches the model weights. It iteratively edits only the state outside the model — the prompt, tool interfaces, and control logic — using execution feedback. The paper likens it to "training a model without updating its weights."
Pass@1 only counts when you get it right on a single attempt. Unlike Pass@k, where getting it right in just one of several attempts is enough, it's closer to the real-world reliability of nailing it in one shot.
Both are multi-step, long-horizon benchmarks that require many tool calls and correcting execution along the way. A harness proves its worth exactly on tasks like these, not on one-shot queries.