There's a benchmark that gathers up the AI agents that analyze microscope images into numbers and makes them compete. It's called BIABench. Here, an "agent" means an AI program that, once a person gives it a task, walks through the several stages of an analysis on its own and hands back a result. A benchmark is the shared exam bench that lines those AIs up against the same test paper and grades them. What this study works hardest to show isn't who came first. It's whether you can trust that ranking at all.
The researchers ran each agent on the same task three times. Common sense says the three results from one agent should look alike, and that entirely different agents should spread further apart. The result was the opposite. The spread among one agent's three scores was wider than the spread between different agents.
Put another way: the same cook, boiling the same recipe three times, got soups that were more inconsistent in seasoning than the soups made by completely different cooks. When that's the case, a ranking you measured once is worth nothing. First place in one round can be dead last in the next.
Don't take this the wrong way. It doesn't mean the leaderboard is meaningless as a whole. It means: don't trust a ranking you measured exactly once. To line agents up and compare them, you have to run the same task many times and look at how far the scores wobble, too. There's one thing this piece will hold onto all the way through: that "getting the same result every time" matters more than a single success — and that it matters just as much who independently checks that result.
Turning microscope images into numbers is harder work than it sounds. You count cells, separate nucleus from cytoplasm, and track a single cell over time. It takes dedicated software and programming, and many labs don't have those hands. So the picture of "an agent that analyzes and reports on its own if you just hand it the images and the instructions" is tempting.
The problem is that no one had properly measured whether that picture actually works. Existing benchmarks looked at just one slice of the analysis — whether it segments nuclei well, whether it tracks cells well. But a lab's analysis doesn't end at one stage. You clean up the image, stitch the tiles, segment (segmentation — tracing out the boundary of each individual cell in the image), pull features from that, and run statistics before a biological conclusion finally emerges.
BIABench's design is one line. Take an already-published study and split it into three pieces: the original images, the instructions a biologist wrote saying "measure this," and the result the paper reported. You give the agent only the first two and hide the answer. Then you compare the agent's answer against the original result, which passed peer review (a procedure other experts verified beforehand). Sixteen studies became tasks this way. They span many kinds of images, from tissue stains to single-molecule microscopy, and six species of specimen, from bacteria to humans. In short, it's a deliberately tricky mix, built so that no ordinary agent can get through on just one or two tricks. The real longevity of this design is that you can keep attaching new papers as tasks the same way.
Two things are especially a wall for agents. One, the data is so large and complex you can't take it all in at a glance. High-resolution 2D, 3D volumes, and time series are mixed together, so the agent has to grope through it with code. Two, it isn't a problem with a fixed answer — it's an open task where the agent has to decide "what to analyze and how" in the first place. We'll meet these two walls again later.
There are six contenders. A tool-style agent that writes and runs Python code directly (Biomni), three command-line agents built for coding, and two GUI agents that directly operate the screens of analysis programs like Fiji/ImageJ. All of them were fitted into the same frame: given a read-only input folder and a write folder, left to run until they stopped on their own or ran out of their time budget, then graded on the files they left behind.
The clever move in the design is pulling apart the "shell and the brain." Here the shell is the frame that drives the agent, and the brain is the AI model that sits inside it. Even the same brain can have quite a different success rate depending on which shell it sits in. So first they fixed the model to one and ran all six; then they fixed the frame and swapped in different models. They also gave two kinds of instructions: a light one where the biologist only wrote "measure this," and a detailed one where an expert also sketched out a recommended pipeline (a workflow that strings several analysis stages together in order). The implementation was left to the agent. Pulling things apart like this is what makes the later experiments possible.
An agent can run an analysis all the way through, produce a plausible-looking figure, and still land on the wrong answer. So the grading was split in two.
One is the outcome score. You compare what the agent produced against the ground truth — how much the cell shapes overlap (Dice), how well the paths used to track cells line up (SEG, TRA), how similar the distributions of the extracted numbers are (the KS statistic), and so on, all converted to a score between 0 and 1. The names differ, but the principle is one: turn "how close to the answer" into a number. Produce no output at all and it's a 0.
The other is the process score. The stages — loading, preprocessing, segmentation, measurement, statistics, reporting — are laid out, and each is checked for whether it was done properly. An AI judge that reads images does the grading: it looks at the code, logs, figures, and tables the agent left behind and marks each stage pass, fail, or indeterminate. The ranking, though, is based only on the outcome score. The process score is a supplementary indicator for seeing "how it was done."
Can that judge be trusted? They measured this separately too. They picked 20 runs spanning several agents and tasks, and a cell-biology expert re-graded them stage by stage with the judge's verdicts wiped clean. Looking at how much the expert and the judge agreed (Cohen's kappa, the agreement between two graders), they lined up well on the stage of understanding the input and diverged most on the stage of choosing tools. The judge was about two points more generous with passes than the expert. So the benchmark fixes the judge to one and always writes this expert-agreement figure right next to the process score. The point: since each judge has its own bias, don't directly compare the absolute values of the scores against another benchmark.
Performance split cleanly along the shape of the task. The plain two-dimensional tasks were handled well. But when depth (three dimensions) or time was added on top, tasks appeared where performance fell off a cliff. In the experiment that fixed the model to one and compared all six agents, on some of the hard tasks with added depth or time, no agent could clear 0.19 (the per-task average of three runs).
To feel what that number means, recall the grading metrics from a moment ago. Most live between 0 and 1, and 1 is close to the answer. 0.19 is rock bottom. The important thing is that this isn't the score of "the worst agent." On that task, 0.19 was the ceiling that even the best-performing agent couldn't get over.
What's more troubling is that the attempts to lower this wall rarely worked on the hardest tasks. Using an agent specialized for biology didn't help, and neither did handing it detailed instructions with the pipeline drawn out by an expert — the wall stood. The experiment that swapped in a more powerful model split by task: 3D zebrafish segmentation climbed to 0.84 and was essentially solved, but the nuclear-pore quantification task stalled at 0.25 and stayed at the floor. In short, it's not that everything collapses the moment depth or time is added — rather, certain hard tasks remain walls you struggle to clear even pulling all three levers. Why depth and time specifically? The two walls from earlier overlap here. 3D and time series don't fit in a single glance, so you have to grope through them with code, and the agent has to devise that groping strategy correctly on its own.
A low score, fine. What's more unsettling is that "a low score" and "the agent believing it did fine" move independently of each other.
The researchers took runs that were submitted but scored low and sorted them again by whether the final message said "done." The failure cases in the appendix are vivid. One agent said "I've finished the analysis" and submitted a table of cell counts — yet the numbers in that table contradicted the explanation in the body of the report. In another case the pipeline looked fine on the surface, but the threshold that decides whether a spot (puncta, the small dots that show up in fluorescence) "counts as a dot" was never once calibrated to the end, so the result was wrong top to bottom. One run declared completion in its closing message in the future tense — "I will now perform the analysis" — having done nothing at all.
Here's the decisive part. Without knowing the answer, you couldn't tell the runs that got it right from the ones that got it wrong by their process score, nor by the time they took. Whether the process was written up cleanly or a lot of time was spent, none of it connected to whether the answer was correct. One more thing: there were even runs where the request to analyze a microscope image tripped the provider's safety filter and was refused outright.
Picture a situation where an AI agent submits its deliverables along with a "analysis complete" report. It's the same whether it's an automated analysis pipeline a company has installed or an in-house AI assistant. We tend to take this one line, plus the attached result file, as proof that "the work is finished." There's an output, and the agent said it was done.
BIABench pulls the ground out from under exactly that one line. What the failures we just saw are telling us is one thing: neither the word "done" nor the existence of an attached file is proof the answer is right. Whether the process is clean, how much time was spent — same story. And yet those are precisely the signals we reach for when deciding whether to trust an AI: self-report, the tidiness of the logs, the time elapsed.
Our team (ONTLE) is redrawing the criteria for how an AI PM judges "done" from exactly this angle. What has to change isn't getting rid of the word "done" — it's the structure where that word becomes the grounds for passing. We detach the completion verdict from the agent's self-report and hang it on an axis that independently verifies the result itself. We don't throw away the process record. We keep it as explanatory material to show the customer "how it was done." But the "pass" stamp only goes on when the result meets the bar.
There's one snag. Customer tasks often have no answer to check against right now. So we pull the moment of verification forward, to the moment we take the job. Say a shopping mall hands us the job: "From last month's order data, pull the products with high return rates." At intake, we nail down three things together with the customer: a sample answer (a handful of top items the customer already knows by hand), the figures to be reproduced (the formula for the return rate and the total number of returns last month), and the minimum bar to pass (that the sample items all land near the top of the result and that the aggregate total matches the customer's books). When the AI PM declares "done," the result goes to the customer only after the system automatically checks it against these three. On top of that, we run the same data several times and check that the list comes out nearly identical every time. The fact that it got it right once is no guarantee it will the next time.
The authors name two limits of their own: the performance gap and reliability itself. On depth-and-time tasks, today's agents are far from experts, and while they cite data complexity, reliability, and a lack of self-checking as causes, they write that the fix is still an open question.
The honest loose ends of the experiment are worth noting too. Two open-weight models (models whose internal settings are public, so anyone can download and run them directly) couldn't be evaluated and were dropped, because in one coding frame an "empty API response" was wrongly logged as a "completed turn." Some runs of the wound-healing task hit an image-size limit and were excluded, then re-run until three valid runs accumulated. Paradoxically, these compatibility issues support this paper's very claim that "the frame shakes the result as much as the model does."
I'd add a limit of my own. There aren't many tasks. In trying to cover a wide range, the sample thins out once you slice it by condition. So this study's conclusion holds as an "alarm" but is still too early to read as a "verdict" on which model or approach is better. Also, GUI agents that operate the screen leave almost no code, so the share that drops to "indeterminate" in the process score is large. The benchmark pulls these out of the denominator and writes them down separately under an auditability (the degree to which you can trace and confirm after the fact how a result came about) indicator, but it shows how hard it is to fairly line up different interfaces under a single process score.
Yes, released together. The arXiv version is under CC BY 4.0. The original study data are all openly licensed materials as well, and are redistributed with the benchmark under each one's original license terms.
It wasn't blocked. But auditing all the stored trace logs showed the network was used only to download packages and model weights (the bundle of internal numbers a model learns during training), and to open software documentation twice.
Every instruction forbids searching for the original publication and its reported figures, and identifying which paper it is from the filenames or metadata. Browsing general methods literature and software documentation is allowed.
The NF-κB translocation task (the phenomenon where, when a cell is stimulated, a protein called NF-κB moves into the nucleus), whose reference values can't simply be left in place, is an exception: it's scored by the dose-response behavior the submitted table shows.