Picture a drilling operator. The VP wants a "generative-AI success story" to show the board in November. He asks for two things: a short summary email every morning, and an "Ask your wells" chatbot. And he draws a line — the downtime management system already exists, so it's out of scope this time.
Build precisely what was asked and, on the surface, it's flawless. It ships on time and passes every functional check. Yet the customer's business metrics don't budge an inch.
The real bottleneck was somewhere else. When downtime turns out to be a contractor's fault, this company is owed compensation from that contractor for the lost time. It's money they should collect every month. But that money keeps getting stuck, because an aging auto-classification script misattributes some downtime to the wrong contractor. Any case with the wrong party assigned becomes a dispute, and because compensation claims are bundled and filed a month at a time, a single tainted case freezes the entire month's claim. This happens at least once a month. The VP's summary email and chatbot have nothing to do with this loss.
To stop the loss, you have to reopen the very downtime management system the VP declared "out of scope." Fixing the misclassification means finding the contract rules that apply, getting them confirmed by the cost engineer, and securing change approval from an approver who happens to be out of office that week.
This is where the gap between the request and the result opens up. Building exactly what the customer said is not the same as actually moving the customer's metrics. This doesn't mean distrusting the customer. It means you have to first work out whether the request carries the constraints that actually decide the outcome — or whether it's aimed at the wrong target entirely. And those constraints are usually hidden in documents and data, with no one spelling them out.
The benchmark that puts a number on this gap (a benchmark is a standardized test that measures AI performance under identical conditions) is FDE-Gym. AI agent benchmarks so far have barely touched this "last mile" — the final stretch of delivery. They hand over inputs that spell out "fix this, like this" — GitHub issues, specs, task tickets — and grade pass/fail with hidden tests. The problem to be solved was already fixed in advance. FDE-Gym hands the job of defining that problem to the agent too. The paper calls this field role the "forward deployed engineer (FDE)" — an engineer a vendor embeds inside a client company to work alongside them — and puts that exact situation on the test bench.
The heart of FDE-Gym is how it grades. Instead of "did the task get completed (yes/no)," it scores "how much did the delivered system lift the customer's own success metric." So we first have to look at what the agent receives and does.
The agent receives a workspace containing the task brief, a copy of the operational system, relevant documents, historical data, and staff contacts. Its job is to find the decisive clues deliberately scattered inside — that contact has a cost, that some people must not be contacted, that the request itself may be misguided — and deliver a system that actually moves the metric. What gets evaluated is not a functional pass but the change in the customer's own metric.
Two solutions anchor the two ends of that metric change.
Between them sits the literal solution — the output that implements the customer's words verbatim. On tasks engineered so that a constraint missing from the brief decides the outcome, this solution is capped by design at 0.2 points. (Which tasks are like this, and what the exceptions are, is laid out in the Appendix below.)
On top of this there's a cost: the stakeholder contact cost. Whenever the agent messages someone at the client, runs a trial pass, or burns a day waiting for a result, points get deducted. Each person has a "rate per message," and the rate rises if you exceed their patience or violate a contact restriction. It's a device to block the strategy of cornering people and interrogating them endlessly.
The final score is set like this. First, take the performance improvement — how far the deliverable lifted the customer's metric between 0 (existing system) and 1 (expert solution) — and subtract the contact cost. Operational incidents are handled separately. If even one thing that must never happen in the field occurs — like acting without approval — all the positive improvement accumulated up to that point is wiped to 0. Conversely, negative losses that worsen the metric stay on the books.
The bar for a "solved" verdict is high. A task counts as solved only when operational incidents are zero and the final score is 0.8 or above. Since the literal solution caps at 0.2, it falls well short of this bar even before the contact cost is subtracted.
Finally, the researchers held the environment, the people, and the grading criteria fixed, varying only the amount of information given to the agent. Look at just the two ends. One condition reveals the root problem, the rules, and the constraints in full; the other gives only the customer's original request as-is. The latter is closest to the starting point of real delivery, and it is the main evaluation setting. Comparing the two conditions lets you isolate whether the performance difference comes from the difference in disclosed information. (The intermediate stages and their English labels are laid out in the Methodology below.)
Even the leading models lose points at three spots.
First, the information condition. Tell them the rules and constraints from the start and the leading models score 0.73–0.82; make them figure it out themselves and that drops to 0.39–0.62. But where this gap comes from needs unpacking. Looking stage by stage across the same 20 original tasks, the moment the rules and constraints are hidden (L0→L1), the average drops by 0.28. Go further — making the request vaguer or giving only the original request (L1→L3) — and the additional drop is just 0.04 on average. Nearly all of the decline comes from that first step of concealing the rules and constraints. That small L1→L3 gap should not be read as "keeping all the information private has little effect" — the big effect already showed up in the earlier step. Nor do the variant tasks make performance worse. The top three models' L3 scores are actually a bit higher across the full 95 tasks including variants than on the 20 originals (0.55→0.56, 0.46→0.49, 0.33→0.38).
Second, implementation. Even in the request-only condition, leading models extract 72–85% of the key facts held by stakeholders. Yet their actual improvement stalls at 0.60–0.70 of the expert solution. Even across the 23 tasks where they extracted every key fact, only 10 cleared an improvement of 0.8. Knowing and making it work are different things.
Why get stuck after knowing everything? Here's one path. In this environment the operational system keeps calling the "original entry point" — the input to the code that operations actually invokes. So even if you write new, correct classification code, the metric won't move an inch unless you integrate that code into the entry point so operations calls it. In the drilling case, "knowing" the contract rules and having those rules reflected in the classification logic and actually applied in operations are entirely different things. This is not an observed failure cause, but one possible path to being blocked despite having all the information.
A low final score does not automatically mean "botched the job." Where an agent touched operations without approval and actively worsened the metric, a negative loss stays on the books; but there are also cases where the metric improved while the score fell because the contact cost was subtracted for cornering people too many times. The former is an incident; the latter is the price of delivering results.
Third, the contact cost. As the price of cornering people to produce results, even leading models lose 0.14–0.22 off the final score.
Start with the limitations the authors state themselves. The human baseline is still being collected. On the fde-gym.com online platform, industry experts are solving the same L3 tasks with the same grader under recording conditions, and those numbers will be reported in the next version of the paper. For now we know "how far models fall short of the expert solution," but "how far short of a skilled human" is still blank.
That it is a "simulated customer" rather than "real delivery" is a fundamental constraint. Scripting stakeholder responses for reproducibility is both a strength and a limit. Real people react far more nonlinearly depending on the conversational context. Also, because the 75 variants and the references are kept private, full external reproduction is hard (the 20 originals and the grader are provided on request, with 3 released publicly).
Because the benchmark adapts post-2024 projects from China, Singapore, and the US via industry experts, its industry, regulatory, and cultural distribution may skew one way. That said, the paper de-identifies or fictionalizes companies and individuals and documents the relevant regulations, standards, and public sources. One more thing: when averaging scores, a correction sets a floor so that extreme negative losses don't dominate the mean (this does not apply to single-run scores). Keep this correction in mind when reading averages.
ONTLE is SaaS where the software "deploys" into the field. Rather than sending a human engineer, an agent in the "AI project manager" role takes a work request from the client's point person, wires up the data, writes the code, and pushes it into real operation. The human sits where the request is issued and the change is approved. This setup meets FDE-Gym at three points — confirming the goal a request aims at, getting operational changes approved, and measuring results after delivery. FDE-Gym is, after all, a script-fixed simulation, so what overlaps is these three points, not the whole field. That is why this paper hands ONTLE a list of "product hypotheses to test."
We check against three axes. We don't assume we already have these; they are the questions we'll work through one by one as a checklist.
Of these, the first hypothesis we test like this. Pick one recurring request type and split intake into two channels. One passes the request straight into work as it does now; the other inserts, before work begins, a step that aligns with the point person on "what metric this request is trying to lift." Run both channels over the same period, then compare three things — the number of rework requests that come back after delivery, the customer metric that actually moved, and the extra customer-facing time the alignment step added. If goal alignment reduces rework and moves the metric more while not adding excessive response time, then that step is not a cost but a condition for results.
Here, "configuration" means the combination of a model and a run setting. Even for the same model, running it with extended thinking (a feature where the model reasons longer before answering) on versus off counts as a different configuration.
| Configuration (L3) | Average Score |
|---|---|
| Claude Opus 5.5 | 0.56 (95% CI 0.50–0.62) |
| Claude Sonnet 5.5 | 0.49 |
| GPT-6 Astra | 0.38 |
| GPT-6.1 Sol | 0.37 |
| Others | three near 0.2, two barely above 0, four below 0 |
A confidence interval (CI) is the range within which the true value is believed to lie. The paper calls the top three "leading models."
First-place Opus 5.5's 0.56 is an average score, not a share of tasks "solved." Because the "solved" bar is zero incidents plus a score of 0.8 or above, most tasks remain unsolved even at an average of 0.56. In fact, no model solved more than 15 of the 95, and five of the 13 configurations solved none at all. Even pooling only the "best configuration's score" for each task (the theoretical best) yields an average of 0.70 and just 30 solved, and on 16 tasks no configuration reached 0.5. Even cherry-picking the best across models, a large distance from the expert solution remains.
Low-scoring models can leave the customer worse off than leaving them alone. Claude Haiku 4.5 scored below the existing system (the do-nothing state) on 79 of 95, and Mistral Small 4 fell below it on 51 even before the contact cost was subtracted. On 5 of the 20 scenarios no model cleared an average of 0.5, and in only one scenario, bank_scam_hold, did any model hit an average of 0.8 or above. Pouring in more money and compute (thinking compute) did not raise scores consistently either.
system/ folder) goes into operation as-is with no modification, and operations keeps calling the original entry point (the input to the code that the operational system actually invokes), so a new feature must be integrated into that entry point to work.