Telling a coding AI "that's wrong" is easy. The hard part comes next: figuring out whether it actually fixed the thing, or just went through the motions and moved on.
Opera is a coding critic (a critic watches an agent work and hands it corrective feedback) built for exactly that gap. Instead of flagging a problem once and walking away, it stays on the issue until the problem is actually gone. On long coding tasks—the kind that spend hours reworking an entire repository—it posted a clearly higher success rate than the no-critic baseline. The widest gap was 15 percentage points.
A coding agent on a long task digs through code, edits it, and runs it until the job is done. The longer it runs, the more it goes wrong. It repeats the same failing command over and over, re-reads files it has already seen, and declares "done" with no verification.
A critic that steps in mid-task is the natural prescription. But there's a catch. A critic has to judge work that isn't finished, with only half the evidence in. At this stage, the line between "a genuine mistake" and "a reasonable intermediate step" is blurry. Get it wrong, and the critic derails a task that would have succeeded if left alone.
So what makes a good critic isn't clever criticism. It's the discipline to watch all the way through: when to intervene, whether the critique is warranted, and whether it landed. Existing critics handle the first two. They evaluate work in progress, or grade a finished result. What they rarely watch is what happens after the feedback is delivered. That's the exact gap—flag a problem, never check whether the fix stuck.
This is the gap Opera digs into. It splits the aftermath of feedback into two questions: Did the agent do what I asked? And did the original problem go away? Not conflating those two is, arguably, the whole of what Opera is about.
That's abstract in words, so let's follow one scene.
The agent tries to make a test pass and fails. Opera steps in and diagnoses: it's missing a guard (a defensive conditional that keeps bad values from getting in) to filter out a certain input. Then it writes this problem onto a single note. The note doesn't just record the diagnosis. It also pins down what has to be true before the problem counts as solved. This time: "the originally required test passes."
The note goes to the agent, and the agent adds the guard to the code. Here a naive critic would say "I asked, it did it" and close the note. Opera looks at it in two stages. First, did it follow the instruction—yes, the guard is in, so that passes. But then it asks the more important thing: so does the original test pass now? No, it's still red. The criterion pinned down at the start hasn't been met, so the note doesn't close. It stays open and carries over to the next review.
This is exactly where "going through the motions" and "actually solving it" part ways. On the status report alone, both read identically as "done."
This note is the heart of Opera. The paper calls it persistent memory. It bundles together one diagnosed problem, the guidance to give right now, and a resolution criterion that, once set, never changes. It's just like a comment on a code review that stays open until someone checks it off as "fixed." Because the comment doesn't disappear, the critic can chase the same problem across several reviews, and because the criterion is fixed, it judges by "did the original problem go away?" rather than "what did I just ask for?" Only one note is open at a time. Three disciplines drive this note.
When to look. Reviewing only at fixed intervals is slow to react (failures repeat until the next review), while looking only when something blows up misses silent failures (confidently editing the wrong file). Opera mixes the two. It runs every so many turns, and it looks immediately the moment any of five signals fires: a stall, a repeat, an error, a completion claim, or a final submission. The key point is that looking doesn't mean it has to intervene. If things are going well, the critic passes by without a word. Since needless meddling ruins tasks that would have succeeded, silence is designed in as a legitimate option.
What to flag. Left to write freely, a critic easily lapses into vague lines like "consider edge cases." For the agent, there's nothing concrete to act on. So Opera forces every diagnosis into one of a predefined set of tags. The tags are tied to the five stages of fixing code—locate, review, edit, test, submit. The crucial part is that each tag comes with its own rules: you may raise this critique only when this kind of evidence is present, and you judge whether it's resolved by this kind of criterion—that promise travels with the tag from the start. As a result, critiques don't overlap, and the yardstick for later deciding "is it resolved?" doesn't wobble. On top of that, each pass picks exactly one well-grounded issue and the smallest correction for it. Not dumping a pile of instructions is what keeps them from colliding and lets you trace what actually worked. (There are nine tags in all; the full list is at the end of the post.)
How to filter out wrong critiques. A critic is, in the end, a language model too, so it produces diagnoses that sound plausible but are wrong. So two checks sit over the life of a note. One before it goes out: does the evidence support the diagnosis, does it truly need fixing, and can the proposed check actually prove it's resolved? Cleanup unrelated to the task, and nagging with no new evidence, get filtered here. One more before it closes: is there actually evidence that it's resolved? If the note demanded behavior, it checks the run result; if it demanded a code state, it checks the actual edit—and only then closes. These two checks are Opera's filter.
Opera exchanges only execution logs and verbal feedback with the agent, and never touches the model weights. That's why it drops straight into an existing harness (the execution framework that wires an agent to its task environment).
It was measured on three benchmarks (standard tests for measuring performance) with different characters: Terminal-Bench, which deals with commands in a terminal; SWE-Bench Pro, which fixes real software repositories; and DeepSWE, hard tasks with the lowest baseline success rate of the three. The three were chosen deliberately for their different grains—to see whether this is a framework that works broadly, not a trick that only works in one environment.
The core comparison keeps the same agent (the model doing the coding is held fixed) and swaps only the critic. Gains are written in percentage points (pp).
| Method | Terminal-Bench | SWE-Bench Pro | DeepSWE |
|---|---|---|---|
| No critic | 65.9 | 75.3 | 32.4 |
| SWE-PRM | 68.9 | 77.7 | 36.6 |
| SWE-Search | 70.0 | 77.3 | 38.6 |
| LLM-as-verifier | 71.2 | 78.7 | 38.9 |
| Agentic Rubrics | 71.9 | 78.3 | 39.5 |
| Opera | 73.8 | 79.3 | 41.3 |
Opera is first in all three. But the margin over second place varies. On Terminal and DeepSWE it leads by 1.9pp and 1.8pp, but on SWE-Bench Pro it's only 0.6pp. The reason is in the top row of that same table. On SWE-Bench Pro the no-critic baseline is already strong at 75.3. When the game is already being played well, there's little room left to lift, whatever critic you bolt on.
This "room to lift" is the key that threads the whole result together. The size of the gain was decided less by the model's absolute ability than by how much room was left to correct. It's the same logic as a targeted tutor being most valuable to a student who misses by a hair, rather than one already near full marks. The maximum gains across the four models were 12.4, 15.0, and 8.9pp, and on Terminal and SWE-Bench Pro the weakest model gained the most. Conversely, tutoring is no use to a student with no foundation at all, one who can't solve the problem in the first place. That weak model solved not a single DeepSWE task with or without a critic, so DeepSWE's largest gain went instead to a stronger model. One model nearly tripled its success rate from the floor, but the absolute gain itself was small.
So far the story is "add a critic and it goes up." The part worth looking into further is what produces that lift.
The first answer is the filter. Strip out both of those checks so that every critique is passed through verbatim and every close request is accepted, and the gain collapses. On Terminal, +7.9pp drops to +2.6pp (two-thirds evaporates); on DeepSWE, +8.9pp drops to +4.5pp (half). It means the device that picks only the accurate critiques and holds them until resolved accounts for more than half the gain—more than making lots of critiques does. The higher the price of a bad fix, the larger the filter's share.
The second answer is more surprising. Using the agent itself as the critic still raised the score—even though it can add no new knowledge. On DeepSWE, two models each gained 6.2pp acting as their own critic, matching or exceeding a stronger external critic. Of the 33 critic-swap combinations, only one came out worse than the baseline. At this point it reads as though the large share of the gain comes not from how smart the critic is, but from the framework itself: looking at the right time, diagnosing by type, and chasing until closed.
But we should pause here. Self-critique also spends more inference, and this comparison didn't equalize that compute. So it's premature to credit the entire improvement to the framework. On top of that, if the agent has no capital of its own to solve the task, self-critique gets no traction. The weakest model gained at least 12.4pp on Terminal with an external critic, but at most 2.7pp with self-critique.
The work logs Opera leaves behind as it runs have one more use: they become training data you can use later, even without a critic. The researchers trained a small model on two kinds of logs. One was the small model's own solutions produced under Opera's coaching; the other was worked answers solved by a stronger model.
On one benchmark, both rose about the same. The contest was decided on the other. Having the small model copy the stronger model's answers verbatim actually crashed Terminal success from 23.6% to 7.9%. The side trained on its own logs under Opera's coaching held at 25.8%. Because every action in that data came from the student itself and the critic's touch entered only as a note alongside, it didn't lose its own pace. It's the difference between a workbook the student solved themselves with only the teacher's red-pen corrections, and memorizing the teacher's worked answers wholesale. The former wobbles less.
The researchers nail it down themselves: a critic cannot manufacture ability that isn't there. The earlier "couldn't solve a single problem" case is the proof.
There's a subtler trap, too. A strong model gets rescued more often because it takes feedback well—but at the same time it more often wrecks a task it would have solved on its own. Recovery and disruption are two sides of one coin. It also means the price of needless meddling is larger the stronger the model.
A more fundamental limit is worth noting. As long as the critic is a language model, wrong diagnoses never fully disappear, and the two checks meant to filter them lean on yet another language model's judgment. A filter can't fully vouch for itself. And as we saw on SWE-Bench Pro, when the fundamentals are already solid, differences in critic design barely show.
At ONTLE, an AI PM takes a single customer request and manages the process as an internal worker handles it over several hours. Say the customer asks to change how the refund amount is calculated when an order is canceled.
The PM hands the work to the worker. A moment later the worker reports "done." Stop here and ONTLE's chronic illness flares up again: the fake "done." The worker did touch the refund-calculation code, but no one checked whether the amount comes out right under the exact condition the customer named.
Done the Opera way, this report is mere "compliance." The moment the PM takes the customer request, it pins a resolution criterion onto a note: under this condition, the refund amount must come out like this. Even when the worker says "done," the note doesn't close until that criterion is cleared by an actual run result. Treat the completion claim and the deliverable submission as review signals, and you can hold the criterion up to a fake "done" right before it crosses over to the customer as "complete."
Before the note goes out to the worker, one check sits in front of it: a peer PM cross-reviews whether the critique is grounded and whether the fix is truly necessary. It stops the accident where a plausible but weakly grounded intrusion derails a task that was running fine—before it reaches the customer. The fact that removing this check in the Opera experiments blew away more than half the gain reads as evidence that, in an environment like ONTLE where an outside customer is on the line, this step is not optional but mandatory.
There's a hint about resource allocation, too. The finding that self-critique worked well on capable models reads as: tasks with enough capital behind them need less heavy external verification, and you can pour the effort into tasks that lack it. That's a lesson you can carry straight into PM operations.
Yes. Opera doesn't train the model further; it lifts performance by spending more compute on diagnosis and correction at run time. So the added compute cost rides along with the higher success rate. The success-rate numbers in this post don't show that cost on their own, so when you weigh adoption you have to look at both together.
That's why closing is handled before opening. A single review closes one issue that's fully done and immediately opens the next. There's always just one open, but the flow doesn't stall.
The three benchmarks ran on different execution frameworks, but the tasks were managed through a common tool and the network was isolated to prevent cheating that pulls answers in from outside. That Opera stayed consistently ahead even across different environments backs up that the result doesn't lean on any particular setup.
| Stage | Tag | What it catches |
|---|---|---|
| Locate | repo_localization_search | Hasn't found the thing to fix yet |
| Locate | implementation_target_shift | Fixing the wrong target |
| Review | implementation_readiness_review | Re-reading code when the fix site is already clear |
| Review | failure_signature_triage | Misreading tool output |
| Edit | requirement_contract_review | Requirement violation |
| Edit | diff_scope_review | Missing or unrelated changes |
| Edit | state_transition_review | Flawed execution logic |
| Test | minimal_repro_or_focused_verifier | No evidence to make the next decision |
| Submit | submission_readiness_review | Claiming done without evidence |