Handing a coding agent a request like "build me an agent that handles this task for me" is becoming routine. AgentSWE is the first benchmark to tackle head-on whether what comes back actually does what was asked. Across 25 tasks spanning three categories — generation, editing, and optimization — none of the coding agents evaluated could reliably produce an agent at the level the requirements called for. The gap between the performance shown during development and the performance of the actual delivered artifact wasn't the exception; it was the default.
First, let's separate two terms. There are agents at two layers here. One is the agent being built. For example, a program that reads customer inquiries at an online store, sorts them into refunds, exchanges, and shipping delays, and routes each to the right staff member — a program that decides and acts on its own without a human attending to every case. That's what we call an agent. The other is the agent doing the building: the coding agent (think Claude Code or Codex) that takes requirements in natural language and writes the code for that program. What AgentSWE puts on the stand is the builder; what it scores is whether the thing built actually works.
There's a quiet assumption baked in here: the belief that a coding agent, handed a requirement, will deliver "an agent that faithfully carries out that requirement." The problem is that there was no tool to verify that belief.
Existing benchmarks measure something of a different grain. The SWE-bench family checks the code a coding agent writes against known-good patches and tests, and even benchmarks that build a repository from a spec document or reproduce an implementation from a paper ultimately "grade code against a predetermined answer." Conversely, the agent benchmarks that came after AgentBench take an already-finished agent and only ask how well it performs. Whether the coding agent could have built that agent in the first place — nobody asked.
Other threads in the problem space ask the wrong question too. Research on "agents that build and improve agents" (e.g., Arbor, JIT-Agent) measures how much a special search procedure or meta-agent lifts the score, and agent-construction benchmarks like WorFBench and AIE-Bench look at improvement within controlled bounds. AgentSWE differs here in two ways. First, it puts a general-purpose coding agent on the stand, not a specialized apparatus. Second, instead of score-lifting, it asks about fidelity — how fully the requirement was met in actual behavior: "does it actually deliver the agent the natural-language request described?" And it covers the entire lifecycle — generation, editing, and optimization — not just the single step of generation.
The framing that runs through AgentSWE's design isn't an "exam" — it's an "order." An exam means solving a problem whose answer you know; an order means taking a request a customer voiced in plain words and delivering a product. Throughout this piece we'll stick to this one analogy: take an order, deliver the goods.
Each task starts from a natural-language requirement. The paper splits that requirement into four parts and hands it over: the goal, which states what the agent is for and what satisfies the user; the interface contract, which fixes how it runs and what it must produce; the constraints, which bound what may be modified, installed, or called; and the resources, such as access to data, storage, and models. The principle is to show the builder (the coding agent that makes the agent) only what a customer could actually hand over.
The key mechanism is the separation between "development cases" and "held-out cases" (evaluation examples hidden from the builder). The builder freely pokes at a small set of public development cases while working. But the cases the delivered product will actually face are not written down. Evaluation runs the frozen deliverable against these held-out cases separately. In other words, the builder saw only 2 examples during development, while the pass/fail review runs on separate cases cutting across the full requirement.
Because there's no reference implementation to grade against, every run of the generation and editing tasks is scored with a rubric (a scoring table) drawn directly from the requirement. In front of that sits a hard gate (a minimum checkpoint that scores 0 before anyone even looks at the content if it isn't cleared). If the agent won't even run, if the output can't be parsed, or if it strays outside the permitted scope of modification, it scores 0. Good-looking code that doesn't work earns nothing. Optimization tasks are measured by the target metric of the original benchmark.
This structure splits two things into one number each: the performance the builder showed during development (demonstrated fidelity) and the performance the actual deliverable achieves (delivered fidelity). The paper argues the gap between the two is the key variable eroding trust in delegation.
Three pieces form the skeleton worth holding onto through this section.
First, there are three kinds of tasks. Generation (10 tasks) asks for a complete new agent delivered from an empty workspace. Editing (10 tasks) layers a new feature onto the frozen code of a running production agent without breaking existing functionality. Optimization (5 tasks) pushes a working agent toward a measurable target.
Second, development cases and evaluation cases are physically separated. Generation and editing give each task 2 development cases and 6 held-out cases (150 development and 365 held-out in total), and the builder works within a single session on top of a container framework called Harbor, then submits a snapshot. The submission is scored only against the development cases, summary feedback comes back, and when the budget runs out the artifact is frozen. Evaluation then runs this frozen artifact separately against the held-out cases.
Third, the underlying model of the things being compared is unified. When running the held-out cases, every model call made by the delivered agent goes through a gateway the benchmark owns, to a single runtime model. All deliverables in the same task group share the same model. That way the score reflects the builder's contribution — prompting, tool orchestration, state management, output validation — rather than a difference in underlying model performance.
Before reading the scores, a word on the scale. The generation and editing scores are fidelity scores measuring "how much of the behavior the requirement described did it actually accomplish." Higher is better, and catching the hard gate means 0. Elegance or maintainability of the code is not measured — the judge looks not at the code but at "the output the agent actually produced on one concrete case" (generation used GPT-5.6 Sol as judge, editing used DeepSeek-V4.1-Flash). Optimization is measured not as an absolute score but as how much of the remaining gap to the target was closed. For builders, five models — GPT-5.6 Sol, GPT-5.5, DeepSeek-V4-Pro, DeepSeek-V4-Flash, Qwen3.6-35B-A3B — were paired with the Codex harness (an execution framework that wraps a model to let it use tools and take multiple steps).
Everything below is under the Codex harness.
| Task | Key numbers | How to read it |
|---|---|---|
| Generation (10) | Across 50 runs (10 tasks × 5 builders): mean 11.2, median 3.4, with 20 of them scoring 0. Even the top builder, DeepSeek-V4-Pro, averaged 19.1, with 8 of 10 tasks under 20 points. Only 3 runs cleared 50 — two were formal theorem proving and one was a repository bug fix. | No builder cleared 30 points on more than two tasks — success on one order doesn't carry over to the next. |
| Editing (10) | Even the builders that managed to deliver (DeepSeek-V4-Pro, GPT-5.5, GPT-5.6 Sol) implemented only part of the requested feature. Means of 20–35, with 70 reached exactly once (GPT-5.5 on the Codex task). Qwen3.6 couldn't submit on any task. DeepSeek-V4-Flash delivered on only four. | Adding a feature to a running product fails in two directions: the patch covers only part of the requirement, or nothing gets finished at all. |
| Optimization (5) | No builder, on average, closed more than 5% of the remaining gap. Of 25 runs, 13 came out higher than the starting point, 9 lower, 3 the same. A sign test found significant differences in only 5 runs (the 3 significant improvements were all on BrowseComp; the 2 significant losses occurred once each on τ³ and PinchBench). | Gains depend on the task. On BrowseComp four of five beat the starting point, but on PinchBench four of five came out lower. |
Behind these numbers are three failure patterns. Let's separate the observed phenomena from the suspected causes.
There are also ablations that varied the conditions. The experiment that increased the number of development cases was run on two tasks (PDF and PPTX) with a single builder, three runs per condition, and there the score rose only in the 0→2 range and climbed no further beyond that — but since this observation is limited to two tasks, it shouldn't be read as a general conclusion that adding cases has no effect. Separately, pre-specifying the design actually lowered the PPTX task score, instructing the agent to test only narrowed the variance of the score, and increasing the case budget by 2.5× still left the gap.
The authors acknowledge limitations directly. There's the repetition problem — because builds are stochastic, it's hard to conclude from a single run — and the fact that frontier models were used as judges for generation and editing, so judge-model bias can bleed into the scores.
Worth adding is coverage. 25 tasks — and reverse-engineered from particular open-source agents at that — is hard to call broad. Also, the design of unifying the underlying model within a task group cleanly isolates the builder's contribution, but in a real product the model the builder chose is itself part of the delivered quality. That unification is the right call inside a benchmark trying to measure "the builder's pure contribution," but it should be read with the caveat that it sits one layer removed from real-world product judgment.
ONTLE is in the business of assembling and delivering small agents that take over the repetitive work of small businesses and solo operators. In other words, we stand exactly where AgentSWE put its "builder."
Picture a small online-store owner handing over this: "Handle it for me — sort customer inquiries into refunds, exchanges, and shipping delays and route them to the right staff." ONTLE assembles the agent and runs it on two or three sample inquiries the owner provided. All three are classified correctly. The demo is flawless. What this paper warns about is what comes next. Once it goes live, you get a flood of "a refund request that's also tangled up with a shipping delay," "an inquiry full of slang and typos," "an inquiry with no order number." An agent tuned to only the two or three cases it saw during development collapses here. Layer on a "hollow runtime" that reports the classification as done but never actually routes anything to staff, and the owner receives "complete" for something where nothing happened at all.
So the question the customer needs to check converges to one: not "did the classification result show up on screen," but "was it actually delivered to the staff member." ONTLE's operating principle is already to "verify with real behavior rather than treating a mere liveness signal as completion." Taking AgentSWE as a cue, we propose adding one more step: running the frozen agent separately against a bundle of held-out cases the owner never saw, and not handing it over as "complete" until it passes. It's the final gate before acceptance — one that exposes the gap between "shown performance" and "real performance" as a number.
If you're on the side that builds and uses agents, there are just three things to put into practice.