AI agents are solving the test too fast. When several top models all land in the high 90s, the test stops ranking anyone. You can't tell who's better, or where anyone stumbles.
By "agent" here we don't mean a chatbot that answers a question on the spot. We mean an AI that digs through data, runs its own tools, and gets work done on its own. A benchmark is the shared test we use to measure that skill. The trouble is that the test is fixed. Models improve every quarter, but humans can't hand-author new problems fast enough to keep up. So the test hits its ceiling almost immediately. The paper calls this state saturation. In a human-built materials science benchmark, the agents doing the solving (call them solvers) averaged 95.7, basically a perfect score. There was no room left to see any difference in skill.
AutoSciBench's idea is simple and bold at once. Instead of making humans squeeze out harder problems, it watches how an agent solves a problem, and uses that solving process as a clue to make the problem harder on its own. The test evolves by watching the student.
Benchmark saturation is a headache everywhere, but science is especially nasty. The reason comes into focus when you stack two distinctions on top of each other.
First, LLMs and agents are different. For an LLM (large language model) problem, everything you need to answer is already inside the question, or inside what the model has memorized. An agent, by contrast, has to go out and get the information itself by actually engaging with an environment. It plans, runs tools, looks at what comes back, and analyzes again.
Second, general-purpose agents and science agents are different too. In coding or web tasks, a wrong move throws an error right away. The signal comes instantly, so you can fix it. Science analysis doesn't hand you that friendly correctness signal. A number that comes up mid-task doesn't tell you "this interpretation is right." It's only indirect evidence. For instance, an agent might dig through raw gene-expression measurements to check which perturbation produced an above-average effect, then overlay that result with another piece of evidence before it can even start to reach a conclusion.
Put the two distinctions together and there's only one takeaway. To evaluate a science agent properly, writing a good question isn't enough. You also have to design the data and tools the agent will dig through, and how it weaves indirect evidence into a conclusion. Re-examining raw data, verifying intermediate results, integrating evidence across multiple steps — that's the ability a science task is really supposed to measure. And building even one such problem properly eats up a lot of expert time. Which is exactly why it falls even further behind the pace of the models.
AutoSciBench builds each task in two layers. Think of it like plating one perfect steak.
The top layer is the concept. This is where you decide what to evaluate — the moment you set the menu and say, "today we're cooking steak." Which scientific field, what data it involves, what task to assign — you write down only the ingredients and the goal, and leave the actual recipe blank.
The bottom layer is the recipe. This is where you write down the order in which the dish gets made. And here there's one crucial decision: what do you hand the agent up front, and what does it have to figure out for itself?
The paper's real example is clean. The task is to classify a crystal structure. The ingredient data needed for it (an ionic-radius table, the classification rules) gets handed over as files — but the actual calculation formula and the answer are withheld. The question is a single line: "Classify the structure of this material." The agent has to pull out the data, compute the metrics itself, and apply the rules to reach the answer. The answer key is validated beforehand by the problem-setter running the same calculation. Splitting it into two layers like this means you can keep aiming at the same analytical ability while stamping out endlessly different questions and data.
Up to here, it isn't all that different from earlier work on automatic generation. Prior methods like AutoBencher and BenchMaker also draw problems out of a fixed goal or a quality bar. But they mostly focus on LLM evaluation, where "just write a good question" is the whole job. A science agent, as we saw, has to have its environment, its data, and even the interpretation of indirect evidence designed for it — and the two-layer structure comes straight out of that requirement.
The real differentiator is what comes next. Writing a good recipe is no guarantee the intended dish comes out. The agent can glance at the output of one tool and just guess the answer. In cooking terms, that's a plate served undercooked inside with sauce poured on top to make it look convincing. The paper calls this cheap move a shortcut.
So it adds a tasting-and-fixing step. The finished task is solved independently, several times over, by the solver. A judge (another model, the one doing the grading) watches that solving process and the final answer with its own two eyes. One thing it watches is whether the question and data support a clear, gradeable answer (verifiability); the other is how often the solver actually gets it right across those repeated attempts (solve rate). Picking apart the solving process also reveals exactly where the agent skipped onto a shortcut.
If it's gradeable but too easy, the recipe gets reworked based on the judge's complaints. The data and question are rewritten so that one intermediate output isn't enough — so the task demands extra verification and interpretation. And if the intended reasoning still doesn't show up, the concept itself — the menu — gets swapped out. This fix / re-solve / re-grade cycle runs up to ten times per task.
The last piece is the cookbook notes. It would be a waste to learn something while fixing one task and then throw that lesson away with the task. Left alone, the generator just keeps serving up menus with the same weakness. So the lessons learned during refinement are summarized into experience cards and stacked up. Which design let the agent answer directly; which fix changed the solver's behavior; what homework is still unsolved. Notably, failed recipes get written into the notes too. It's a structure that learns from failure as well.
The most dramatic scene is in materials science. The solver average, pinned to the ceiling at 95.7 on the human-built benchmark, dropped to 70.2 on a benchmark generated by Claude Opus 4.8. Computational biology fell too, from 79.8 to 57.4. The 22.4- and 25.5-point drops the abstract headlines are exactly these Opus 4.8 numbers. The size of the drop varied by generator, but the direction was the same.
More important than the fact that scores fell is that they spread out. Scores that had been bunched up near the top, impossible to tell apart, fanned out widely downward on the generated benchmark. A saturated test started picking out skill differences between models again. That's the core evidence for the claim that this "breaks benchmark saturation."
There's also a signal that this difficulty isn't a fluke that only worked for one particular pairing. Other models that weren't used to build the problems also scored low on the generated benchmark. The difficulty transferred beyond the setter–taker pair to other models as well.
Quality is another axis. The generated tasks scored higher than the human-built ones on average quality across all three fields. Meaning they weren't just hard — they held up as evaluations. That said, the quality score came from two LLM judges — Claude Opus 4.8 and GPT-5.6 Sol — each grading independently, with their scores then averaged. Still LLMs, not human experts — something to keep in mind when reading the result.
The paper also ran experiments that peel the two mechanisms apart one at a time to see whether they actually work (ablations). Iterative refinement raised the difficulty as the rounds went on, and that rise happened in a way that "demanded more re-examination and interpretation of the data." The very character of the required reasoning changed. The experience-based generation — the cookbook notes — reused solving requirements that had already proven effective and built harder problems with them.
The authors admit it themselves. This framework can't produce a good task in one shot. Putting iterative refinement in the loop is itself the proof. On top of that, three more things are worth flagging.
First, the self-reference risk. The generator that builds the problems and the solver that solves them are the same model. The trap of "judging something hard just because you can't solve it" is hiding here. The fact that the difficulty transferred to other models is some defense, but the generator's own blind spots get inherited straight into the tasks. The kind of scientific error the generator misses in the first place never even becomes a problem.
Second, the solve-rate trap. Put it in college-exam terms. A killer item with a low pass rate because it demands complex reasoning, and a defective item with a low pass rate because the passage is ambiguous or has a typo, are two completely different stories. Both show up as a low solve rate. AutoSciBench tries to tell them apart by also looking at verifiability, but there's a limit in that this judge is, again, an LLM.
Third, the trustworthiness of the answer key. The answer to a task is computed by the agent itself. For a calculation-type problem with crisp rules, maybe — but for more open-ended science analysis, whether an automatically produced "answer" is also the answer in an expert's eyes needs separate confirmation.
For reference, the experiments were run in three fields — computational biology, materials science, and clinical imaging — using CompBioBench, MatTools, and MedCTA respectively as seed benchmarks. Note, though, that the clinical-imaging human-built benchmark score was left out of the comparison table. Because it's graded by an LLM rather than by exact string match of the answer, it can't be put on the same yardstick as the other two fields. So the accuracy drop is a story about computational biology and materials science. Don't lump all three together as having gotten equally harder.
This paper's worry overlaps exactly with ONTLE's daily life. ONTLE attaches an AI PM to every customer project. The work a human project manager used to do — take a request, analyze the data, write up the results, and report back — is handled instead by an AI agent. Before we send that PM out to a customer, we ask the same thing every time: "Does this PM actually carry out real analysis on this customer's data, or does it just skate by with a plausible-looking answer?"
The problem is that right now this check leans on a handful of fixed tests. Bump the model up one notch and that test suite passes in full almost immediately. A 100% pass rate is not "it's perfect" — it's the signal that "we can no longer tell the difference" — and we often read that as a passing grade.
Follow one scene all the way through. A customer uploads raw sales data as an Excel file. The ONTLE workspace already previews that file as a table. We tell the PM, "find the anomalous stretch." If the PM runs a tool once, looks at a single summary number that came back, and concludes "a sharp drop in March," that's an undercooked steak with sauce poured on top. Borrowing AutoSciBench's idea, that's exactly the point where we rewrite the verification task. We rework the data and the question so that a single summary number won't settle it — so the answer only stands once you cross-reference evidence across multiple sheets. And if we stack up the shortcuts our PM keeps falling into, like experience cards, we can aim at that weakness from the very start when we design the next customer's verification. The pre-deployment QA turns from a "pass / fail" stamp into "a map of this PM's failures."
One piece of professional advice to add, though. To keep from importing that self-reference risk onto our own side, it's worth thinking about a cross-verification setup that separates the side building the verification task from the PM being verified — using different models for each.
No. The refinement loop runs automatically on the agent, but the starting point is an existing human-built benchmark. Humans also decide which field to evaluate. Rather than creating from nothing, it's closer to a structure that grows out of existing assets and literature used as prior knowledge.
It's best not to see it as a tool that stamps one absolute difficulty on everything. The ground for trust is that the difficulty transferred even to other models not used in generation. In practice, the safer move is to pick a generator to match what you want to discriminate, and always cross-check with an outside model to see whether the difficulty holds up.
The scope this paper validated is only the three scientific fields — computational biology, materials science, and clinical imaging. The core idea of watching the solving process to close off shortcuts seems like it wouldn't care which field it's in, but its effectiveness beyond those bounds is not something this paper demonstrated. To put it to use, you'd have to validate it again in your own domain.