Coding agents don't just write code. They also write the tests that check whether that code is correct. When the tests go green, it's "pass, done." On a real engineering team, that green light is a token of trust.
The trouble is in how that green light gets switched on. The usual move is to run the tests against a single correct implementation and call it a pass if they go green. But for any given requirement, there's usually more than one correct piece of code. And there's more than one piece of plausible-but-wrong code too. Can you really trust a test after checking it against just one answer?
A recent study took exactly this question and re-graded the tests. Instead of a single answer, it ran them across a range of implementations. The pass rate of the top-performing model fell sharply. Not a single character of the tests was changed. Only the way they were measured.
A good test does two things. On buggy code (before the fix) it fails and exposes the bug; on properly fixed code it passes. The industry convention checks both of these against a single answer. Fail before the fix, pass on the gold solution, and that's a pass. This is called single reference evaluation.
There's a trap here. Picture writing a hiring exam tailored to exactly one model answer. If the questions only fit that answer's style, a candidate who got the right result but solved it differently gets rejected. And a candidate who's plausibly wrong slips through unfiltered. Tests are no different. The first case clings to one particular implementation more than it should and breaks other correct code; the second fails to actually check the required behavior. Single reference evaluation stamps both of them "pass."
So the real question is this: is this test trustworthy even when the implementation changes? It should work across many correct implementations while staying sensitive to implementations that violate the spec. The core of this study is that it bakes this old demand into the evaluation metric itself.
The method is simple. Grade the test not against one answer but against an "implementation panel." Each task gets its own panel: 1 reference, 4 differently-written-but-correct implementations, and 5 incorrect implementations. With 10 implementations each across 300 tasks, that's 3,000 labeled implementations. Valid and invalid are split exactly in half. When the test is written, the panel is hidden. The agent writes the test from the requirements alone, and grading is done against the whole panel.
One detail worth noting is how the incorrect implementations are made. Instead of injecting artificial flaws to manufacture fake defects (mutation), they collected the wrong answers that models of varying ability actually produced while solving the tasks. It's a choice aimed at staying close to the mistakes people really make. To keep the panel from getting too easy by piling up similar implementations, they also applied a filter that keeps only the sufficiently different ones.
Grading breaks into four numbers.
The cleverness of this design lies elsewhere. Shrink the panel down to a single reference and JSF collapses back to exactly the old score (RAR). The conventional method isn't wrong; it's the loosest special case of the new yardstick. And JSF can never structurally exceed RAR. The gap between the two is precisely the share of "tests that passed against one answer but crumbled in front of other implementations."
The numbers make the gap plain. Even the top configuration scored 59.67% on the conventional metric (RAR), but only 28.00% on JSF. Tests that passed nearly 60% of the time against a single answer dropped below half once the conditions were added. The JSF of the best configuration per family scatters from that 28.00% down to 11.33%. In other words, "writing tests you can trust even when the implementation changes" is still unsolved, even for leading models.
The details hold an interesting asymmetry. Five of the six configurations did better at rejecting invalid implementations (IRR) than at accepting valid ones (VAR). It's easy to filter out wrong implementations, but harder to accept another correct one as "this is also a valid answer." One configuration had the highest invalid rejection at 86.33% but came in last on valid acceptance (50.33%) and JSF (11.33%). Being good at filtering alone doesn't make a good test. Conversely, another configuration accepted valid implementations well at 77.00% but was relatively dull at rejecting invalid ones (66.00%).
Let's be clear about one thing. JSF is a score against that particular panel. It isn't an absolute guarantee over every correct and incorrect implementation in the world. In fact, when they picked out tasks and grew the panel, a score that averaged 49.33% with only the reference fell to 24.33% once 9 more valid and invalid implementations were added alternately. The tests stayed the same. The stricter the panel, the lower the score. That's why "JSF 28%" should be read as a measurement under this configuration, not as an absolute grade.
Concrete scenes beat abstract categories. Two examples from the paper are vivid.
Consider code that evaluates poker hands. One test never once pitted an "ace-low straight" against "three of a kind." With that comparison missing, an incorrect implementation that misclassifies an ace-low straight as a high card sails right through. The test simply never exercised that behavior. This is called a coverage gap. Coverage refers to whether the test actually ran the relevant lines of code.
A test for a bracket parser (code that interprets whether brackets are balanced) is a different flavor. It did feed in mismatched brackets as input. But it only checked the count of the result. As a result, an implementation that discards an entire valid bracket group gets accepted as "the count matches, so pass." It reached the code but got the judgment wrong. This is called a wrong assertion. Here "assertion" refers to the single line where the test decides whether the result is right or wrong.
Both tests fail on the pre-fix code and pass on the correct implementation. The conventional method lights up green just fine. Only when you attach the panel does the defect surface.
Now put numbers on it and the meaning sharpens further. When an expert classified 286 failed tests, the "wrong assertion" we just saw was the largest at 50.0%. "Coverage gaps" were 34.6%, and bugs in the test itself were 15.4%. The third kind goes like this: a function is defined inside the test and called, but an extra argument gets passed in, and the test breaks even on the correct implementation.
Here's what to take away. Half of the failures are cases where the code was reached but the judgment was wrong. Coverage only tells you "did it reach the relevant code"; it does not guarantee "did it reach it and then judge correctly." That's exactly why stamping something done on the strength of coverage numbers alone is dangerous.
Now that we have a yardstick for measuring, let's look at the side that pushes the score up. The core of the generation-and-verification procedure the researchers proposed fits in one line: don't write a test once and stop; make several differently and fix them by pitting them against each other. It runs in three parts.
First, several agents, each given a different search strategy, write their own tests. The strategies are made deliberately different because there's more than one path to failure. You have to probe from different angles to hit different defects.
Second, a test written by one agent is run against code fixed by another agent. If my test flags someone else's perfectly good code as wrong, that's a signal the test is overly tied to a particular implementation. That counterexample is used to fix the test. On top of that, an independent reviewer who saw only the requirements separately checks whether the judgment is correct.
Third, the refined "generation strategy" itself is tuned once more on training tasks, and then that strategy is frozen and deployed in the real evaluation.
The most interesting result comes from the ablation experiment that removes components one at a time to measure their contribution. Merely increasing the tests from 1 set to 3 sets barely moved the score. But once cross-verification was switched on, pitting those 3 sets against each other to fix them, the score jumped sharply. In both cases the compute available per task (the inference budget) was held equal. In other words, the gain came not from "generating more" but from "verifying against each other." The complete procedure was 8.67 to 9.00 percentage points higher than the model's default approach, and 11.67 to 12.00 percentage points higher than the simplest single-set configuration. These two numbers have different baselines, so they shouldn't be added together.
There's one honest trade-off too. When tuning the strategy, they targeted the old metric RAR rather than the flagship metric JSF. JSF is so strict that almost no training signal emerges. The researchers themselves acknowledge the mismatch between the metric they measure by and the metric they optimize toward.
The reason this study doesn't read as someone else's problem is that the way ONTLE works is precisely this picture.
Let's follow one scene all the way through. On a customer project, an AI PM takes on a payment refund calculation feature. The AI writes the code, writes a test to verify that code in the same flow, and once the test goes green, reports to the customer: "refund calculation complete." This green light is exactly single-answer grading. As we just saw, its quality may be systematically inflated. It means the "complete" badge handed to the customer may rest on a test that breaks an equally correct alternative implementation, or turns a blind eye to a subtly wrong one.
So the place to change is the review and QA gate in the ONTLE pipeline. We insert a small procedure before stamping complete.
We should guard against hasty equivalences. What this study addressed is the problem of measuring the quality of "agent-generated tests." Not every completion verdict in ONTLE has the same structure. There's no basis for lumping hand-written tests or tests the customer already has into the same number. The cost isn't free either. Each additional implementation means re-running the tests from scratch. That's why it's right to keep the panel small and the gate narrow. Even if the customer screen still shows only "complete," whether the gate behind it finished verifying against multiple implementations rather than a single answer is what makes the trust real.
No. Coverage showed a positive correlation with JSF. Reaching the relevant code clearly helps. But "reaching" and "judging correctly" are different things. It means coverage is a necessary condition, not a sufficient one. Rather than throwing coverage out, the right move is to add a layer on top of it that measures "once reached, is the judgment correct?"
The upside is realism. You get a defect distribution close to the mistakes people and models actually make. The downside is that the distribution ends up tracking the error tendencies of the models that produced the wrong answers. Defects of a kind those models rarely produce show up less in the panel. Since the tasks were drawn from several public benchmarks, it also inherits some of the biases of those sources.
Pick the one feature that breaks most often. You don't even need to write new implementations: reuse code that was actually wrong in past bug reports as "invalid implementations," and use past correct refactorings as "valid implementations." A panel of two or three is enough. The point isn't scale; it's running verification that steps beyond a single answer even once.
There's no perfect guarantee anyway. A panel doesn't hand you "absolute safety"; it's a device that widens the basis for the green light from one answer to many. Rather than seeing trust as 0 or 1, it fits practice better to treat it as a dial you turn to match the risk level. Wide panels for money-critical features, narrow ones for lightweight changes.