Try telling an image-editing AI, "turn this photo into a nighttime scene." What comes back usually looks convincing enough. The trouble starts right after. Did the model actually understand "night," or did it just crank the brightness down and drop a blue filter on top? You've met people who speak fluently but answer a question other than the one you asked. Today's editing AIs are a lot like that.
Gather 58 of the latest editing AIs and have them compete head to head, and even the best performer earned a perfect score across every evaluation axis on just 56.6% of tasks. That barely clears half. What's worth noticing is where they got stuck. For the leading models, image quality and fidelity to the original held up better than their grasp of the instruction. The thing holding them back was the ability to reason about what an instruction actually means and follow through — of the three axes the benchmark scores, instruction reasoning was the one that kept the top model short of a clean sweep.
The source of these numbers is RISEBench++, a benchmark (a standardized test that pits many models against the same yardstick) built by researchers across six institutions, including Shanghai Jiao Tong University. It measures "the ability to truly understand and edit correctly" as something separate from image quality and consistency, across 1,000 tasks. It grew out of a simple complaint: there was no fair way to measure this ability.
Image-generation models have gotten good at two things over the past few years: understanding pictures, and creating new ones. But faced with "fix this existing picture," they still stumble.
What the researchers zeroed in on is a capability one rung above simple retouching. They call it RISE — Reasoning-Informed viSual Editing. It's the ability to read the context, think it through, and edit intelligently: changing the lighting to match the time of day in a scene; adding or removing an object without breaking its meaning; inferring and acting on intent the user never spelled out. Leading commercial models (the GPT-Image family and the like) far outpaced open source on this kind of work, while traditional editing models that don't reason couldn't even get a foothold.
The problem was that no one had a proper way to measure this ability. Many existing editing benchmarks deal with tasks where the job is spelled out — "make the sky blue," "put a hat on him." They test how cleanly a model paints what it's told to, but they sidestep the ability to reason about why an edit should be done a certain way. RISEBench++ aims squarely at that gap.
RISEBench++ splits "reasoning" into three types by character: reasoning that draws on knowledge, reasoning that reads clues from the image, and reasoning that uses imagination.
First, knowledge-driven. Apply the right piece of common knowledge and the answer falls out. Asked to draw "this tree ten years from now," the single fact that trees grow is enough to solve it.
Second, perception-driven. You first have to read the spatial structure or a hidden rule in the picture with your eyes, then reason on top of that. You can't just talk your way to the answer by guessing — you have to anchor yourself in the clues the image provides.
Third, imagination-driven. It borrows common sense but deliberately inverts it. You have to invent a scene that's plausible yet impossible in reality, without the pieces contradicting one another.
To measure these three characters more precisely, the benchmark splits them into six concrete dimensions and designs tasks accordingly. Knowledge-driven breaks into time and causality; perception-driven into space and logic; imagination-driven into counterfactual (positing a situation contrary to reality). On top of that sits a hybrid category that mixes several kinds of reasoning across several turns — a long-haul task where you edit, then edit again, getting each intermediate transformation right while holding the scene's consistency together all the way through.
The scale is clear. All 1,000 tasks were built by hand. The four knowledge- and perception-dimension categories have 200 each; imagination and hybrid have 100 each. The six dimensions are then split further into 65 sub-types. It's a design meant not to produce a single blurry average but to carve ability into small enough pieces to pinpoint where things fall apart.
This is where the benchmark's real design shows. Instead of mashing the result image into one score, it splits scoring into three axes: Did the edit carry out the reasoning the instruction demanded (instruction reasoning)? Did it preserve the original's appearance consistently (appearance consistency)? Does the result look plausible (visual plausibility)?
Why bother splitting it? Because a single combined score misses the most dangerous failure — the one where everything looks fine but the instruction wasn't actually followed. If the appearance is clean and the image is natural but the instruction reasoning is wrong, that edit is, in effect, a failure. Score the three axes separately and that gap shows up as a number.
Scoring uses human judges and an AI judge together. The "AI grading AI" setup usually draws skepticism. The researchers tried to dispel it by separately measuring how closely the automatic scores track those of human experts — in other words, they first verified that the evaluation framework itself is trustworthy.
Where this benchmark stands comes through clearly in the comparison table. RISEBench++ is the only one that has all four: reasoning, multi-turn, multi-image, and agent support (multi-turn meaning editing over several turns — edit, then edit again — and multi-image meaning feeding in several pictures together).
| Benchmark | Reasoning | Multi-turn | Multi-image | Agent support | Task types | Samples | Models evaluated | Language |
|---|---|---|---|---|---|---|---|---|
| KRIS-Bench | ✓ | ✓ | 22 | 1,267 | 10 | EN | ||
| WiseEdit | ✓ | ✓ | 4 | 1,220 | 22 | EN, ZH | ||
| RISE-Bench | ✓ | 16 | 360 | 9 | EN | |||
| RISEBench++ | ✓ | ✓ | ✓ | ✓ | 65 | 1,000 | 58 | EN, ZH |
The RISE-Bench in the table is this work's predecessor. RISEBench++ is the expanded version: it adds multi-turn, multi-image, and agent support, reworks what were four reasoning categories into a six-dimension system, and grows the task count from 360 to 1,000. The predecessor was selected for an oral presentation (top 0.35%) in the NeurIPS 2025 Datasets & Benchmarks track.
The researchers evaluated 58 of the latest editing approaches with RISEBench++: 34 open source, 19 proprietary commercial, and 5 agent-based (methods that don't stop at a single model but wrap planning and verification steps around it). A side-by-side comparison this broad, in one place, is itself rare.
The scores trace an arc of progress. Early open-source models were weak at reasoning and struggled broadly; more recent open-source work has closed much of that gap. Even so, commercial models were generally stronger. One of them took the overall top spot — and that "barely clears half" score mentioned earlier is its.
What matters is where they get stuck. For the strongest models, appearance consistency and visual plausibility tend to score higher than instruction reasoning, so the relative bottleneck sits on the reasoning side. But a gap between axes isn't a solved problem: even leading open-source editors land only in the low-to-mid 60s on appearance consistency (Qwen-Image-Edit-2511 at 64.0, FLUX.2 Dev at 57.7 on the official results table), so drifting away from the original is still a live failure mode rather than a closed case. Instruction reasoning, by contrast, remains the big challenge — especially in logic reasoning and hybrid. Both are tasks where you have to chain several steps together and keep the scene's pieces aligned the whole way through. On the side that demands sustained thinking, the models collapse.
And here's the one-line takeaway of this piece. The weakness of today's editing models isn't "can't draw." It's closer to "draws well but draws the wrong thing." That diagnosis is especially useful for anyone building a product, because it determines whether you invest in polishing the surface of the output or in machinery to verify the instruction was actually understood.
The paper's second thread offers a hint on that verification side. Rather than retraining the executing model to be smarter, the researchers brought a method that wraps a not-so-smart executor (the model that actually renders the image) in "planning" and "verification." It's called RISE-Agent, and it needs no extra training. A planner reasons about the instruction and sets a goal, an executor turns that goal into an image, and a verifier looks at the result and decides whether to accept it, refine it further, or scrap the plan and start over. These three roles cycle round and round. With zero retraining, it still outperformed most commercial models and existing agent-based methods. The lesson: "a bigger model" may not be the only answer.
Let's follow one scene ONTLE's AI PM actually runs into, all the way through. A client asks: "Please change this campaign banner to an evening mood. Keep the product tone from last time as it is." There are three unstated conditions tangled up in there: inferring the time of day from "evening," keeping the product's appearance consistent, and referencing context in "last time's tone."
The AI PM hands the request to the executor and gets a result back. The image comes out clean. That's exactly the trap. It's clean — but it may have mistaken "evening" for nothing more than a dark sunset filter, and it may have ignored "last time's tone" entirely. If we call this "done" and hand it to the client, a good-looking wrong answer slips through. This is the failure we guard against most: the AI reports it executed the instruction flawlessly, while the client's core intent gets missed.
This paper hands us a lens to diagnose that trap: rate the three axes separately, and make the pass criterion "instruction fulfilled" rather than appearance or plausibility. Ported to ONTLE, it looks like this. If the verifier can't pass the instruction reasoning, that task does not advance to "done." It reverts to "in progress," or — when human judgment is needed — flips to "awaiting a decision" and sends the question back to the client. Keep verification as an independent step, separate from execution, and set its yardstick to instruction fulfillment, not image quality. Those two things are the heart of it.
One more point. Rather than retraining the model from scratch to lift performance, thickening the plan-and-verify loop first may be the cheaper route. But it isn't free. The verification step also costs calls and time. Which tasks get wrapped in this loop is a call to make deliberately.
Two things to keep in mind when reading the numbers. First, 56.6% is a strict metric — it counts only tasks that earned a perfect score on every applicable axis — so the variance across task types is large. You have to first see which type your service's work falls into before that number means anything. Second, automatic evaluation ultimately leans on a judge model. It was reinforced with human alignment, but in principle there's still room for the rankings to wobble if you swap in a different judge.
The number 58 is also a snapshot at this paper's moment. Editing models keep pouring out, so the structural finding — "which types everyone gets stuck on together" — lasts longer than any individual ranking. The authors themselves point to this bottleneck as both a weakness and the key direction for future work.
Yes. The code is in the VisionXLab/RISEBench repository, and the dataset is in the VisionXLab collection on HuggingFace. The specific paths follow from the arXiv and HuggingFace links in the references below.
No. Each task is provided in only two languages, English and Chinese. There's no Korean, so it's hard to say from these results alone how things would behave applied as-is to a Korean-language setting.
No. It's a benchmark limited to image editing. Video is outside the evaluation scope.