The truly annoying part of fixing an image isn't how to change it — it's putting into words which thing to change. VibeEdit moves that job from the mouth to the hand. You mark the target directly with a circle, a scribble, or an arrow, and next to it you jot a short note like "make it darker" to say what you want. The mark points to "which one," the note says "what." The authors call this bundle of markings a canvas instruction — an edit directive drawn right onto the picture.
A single bundle covers five kinds of edits: adding an object, removing one, swapping one, adjusting color and material, and moving something. No long descriptive sentences needed.
Let's start with the results. On a 419-item test the authors built, VibeEdit scored 79.9 on overall edit quality (did the edit happen as requested?) and 32.8dB on source-preservation fidelity (did it leave untouched what it shouldn't have touched?). FireRed, the strongest competing editor — which was additionally handed text spelling out the target's location and the request — managed only 67.4 / 24.0dB on the same test. The side given nothing but marks on the picture beat the side given a helpful written description on both metrics. (That said, the test and the scoring prompts were built by the authors themselves — though the judge itself was a separate external model, not one they designed.)
And this problem isn't unique to image editing. It's the same "not this one, that one" problem an ONTLE PM runs into every time a client sends in a revision request.
When you edit an image with text, the hard part isn't the how — it's the which. "Make the color red" is easy. The trouble comes when there are five similar-looking chairs on screen. To single out one of them with words alone, you have to describe its position and surroundings at length — "the third chair from the left, the one turned halfway toward the window" — and even then the model will happily paint the wrong chair.
If you could just point a finger at that chair, all this effort disappears. Pinning down space is something direct marks like dots and strokes do well; saying what change you want is something language does well. VibeEdit bundles the two together, right on the image, in one shot. A circle points at the target region, a scribble means delete, an arrow means move. The short note beside it writes out the desired content or attribute. The key point is that these marks don't have to trace object boundaries precisely. A rough stroke is fine.
The idea of "instructions on the picture" isn't new in itself. Point-dragging approaches (the DragGAN / DragDiffusion family) place handle points and target points to push and pull objects, but the range of edits they can handle is narrow. Approaches that mark only the location and take the "what" as text (mask-, box-, and scribble-based methods like Inter-Edit, FineEdit, and VIBE) still leave the meaning to be carried by writing. There are also systems that pair brush manipulation with an edit prompt (the MagicQuill family). What sets VibeEdit apart is that it puts the "what" onto the canvas too, as a short handwritten note. So the mark and the note act as one bundle, specifying "where" and "what" at the same time. The authors see this not as replacing text, but as supplementing it with a spatial instruction precisely in the moments when words get cumbersome or ambiguous.
The idea of a canvas instruction is intuitive, but the implementation is tricky. The model has to read the marks and apply them to the edit, while never drawing those marks into the output image. If you feed a photo with a circle on it straight into the editor, the editor can't tell whether that circle is an instruction to be erased or part of the picture. The original content and the instruction marks are mixed on one sheet. With mask-based methods that marked only location as a black-and-white region, this mixing was less of an issue — but in VibeEdit, which puts even handwritten notes onto the picture, the distinction immediately becomes the core challenge.
This paper's key mechanism is layer-decoupled conditioning — feeding the original and the instruction marks in as separate layers. Think of a proof sheet. When an editor writes "set this word in bold" in red pen, the typesetter reads the mark and typesets accordingly, but must not leave red-pen marks in the printed book. VibeEdit divides the roles the same way. On one side, an AI that understands images and text together interprets the marked-up photo and extracts "what the instruction is asking for." On the other side, a generative AI that actually paints the pixels receives the clean original and the marks as two separate layers. The marks are drawn separately on a neutral gray background and fed in as their own layer. This lets the model grasp the meaning of the marks while always holding the untouched original as a reference point.
Once the structure is set up, you have to teach it. Training happens in two stages.
Stage one teaches by example (supervised learning). In local edits — edits that touch only part of a photo — most of the photo stays the same, so if every pixel is treated equally, the small part that should change gets drowned out. So a larger weight is placed on the spots where the edit target and the marks are, pushing the model to focus on "where the change happens."
Stage two rewards edits done well (reinforcement learning). Some results look plausible but touch the wrong object or alter a perfectly fine background, and stage one alone doesn't catch these well. So three things are scored and rewarded separately: did the requested edit happen in the right place (completion), is the surrounding area unchanged (preservation), and does the changed part look natural (quality). Preservation in particular gets a thresholded safeguard. Only when the fidelity (PSNR versus the original) outside the edit region — with a margin added — drops below a 25dB baseline does it dock 0.3 from the reward. It isn't punished just for the outside being slightly different, only when it visibly collapses below the baseline.
The whole model isn't retrained from scratch. The underlying editing AI is kept almost entirely frozen, and only a small adapter is bolted on top to teach it just the ability to "read the marks." This efficient way of teaching a new skill by adding a minimal part rather than overhauling the whole large model is called LoRA.
| Setup | Overall edit score | Source-preservation fidelity (PSNR) |
|---|---|---|
| VibeEdit final (marks only) | 79.9 | 32.8dB |
| VibeEdit stage one only (marks only) | 67.8 | 27.0dB |
| FireRed (given a written description) | 67.4 | 24.0dB |
Three things come out of the results.
First, marks alone beat written descriptions. Final VibeEdit scored 79.9 / 32.8dB; FireRed, the top competing model which was handed the target's location and the request all written out, scored 67.4 / 24.0dB. The authors read this gap as a signal that "the cost of pointing at a target is larger than you'd think when done through description." The two approaches differ in their very input format, so it's hard to pin the gap on a single cause — but at the least, it's clear the mark-based approach doesn't fall behind description.
Second, reinforcement learning actually did work. VibeEdit with only stage one (learning by example) scored 67.8 / 27.0dB; adding stage two (rewards) jumped it to 79.9 / 32.8dB. The edit score rose about 12 points and fidelity in the outer region rose 5.8dB. There's one more thing worth noting. Before stage two was added, that 67.8 was already essentially on par with FireRed's 67.4 — which had been given a written description — and the outer fidelity was in fact higher (27.0 versus 24.0dB). The mark-based approach's competitiveness was, in other words, already visible before reinforcement learning.
Third, marks don't get read just because you "drop them in." When canvas-mark images were fed as-is into public editors that hadn't been trained for them, the scores collapsed. The same models jump into the 60s when given a text description.
| Untrained public editor | Fed marks image only | Given a written description |
|---|---|---|
| Qwen-Image-Edit-2511 | 19.8 | 62.5 |
| FireRed | 22.8 | 67.4 |
| JoyAI | 46.4 | 65.4 |
The observed facts stop there. The authors' interpretation is that reading marks is not an innate ability but one that must be learned, and a model that hasn't learned it mistakes the marks for part of the picture — which lines up exactly with why this paper designed layer decoupling.
Broken down by edit type, move was the hardest for everyone. The text-based competitors couldn't get their move scores past the mid-50s, while VibeEdit hit 76.4. The authors see the essence of a move — "from where to where" — as something an arrow, a spatial mark, conveys more directly than language. VibeEdit led evenly across the other four types too (add, delete, attribute, swap; specific figures in the technical details below).
So on what kind of test did these scores come about?
The scoring test set has 419 items, collected and curated by hand so as not to overlap with the training data. The key point is that it deliberately includes many "scenes with several similar-looking objects." The aim is to test not just "does it understand the requested change" but "does it apply that change to the exact one that was intended" — a design aimed squarely at the "which one" problem mentioned earlier. For a fair comparison, the text editors were handed a separately composed description (averaging 21.3 words) carefully spelling out the target's location and the request.
Scores are measured two ways. One is the overall edit score. With an AI that understands images and text together fixed as the judge, it's the pass rate from scoring edit success, surrounding preservation, and local quality on a yes/no basis (this scoring scheme is called a rubric). The other is source-preservation fidelity (PSNR), a pixel-level measure of how identical the area outside the edit region is to the original — higher means less was touched. The two look at different things. The former is "did the edit happen properly," the latter is "did it leave untouched what it shouldn't have touched."
The training data runs to 1.55 million pairs. These weren't made one by one by hand; they were stamped out by an automated pipeline stitching together several existing AI tools. In broad strokes, it's an automated pipeline that finds editable objects, invents a plausible edit request, then filters out low-quality results (tool makeup in the technical details below). The authors themselves admit this pipeline can't filter out every error, and they say the remaining noise is corrected by the stage-two reinforcement learning.
The limitation the authors admit to themselves is the data. Automatically stamped-out edit pairs leave errors behind no matter how much you filter, which is why they correct with reinforcement learning — the design is built on the premise that the training data is noisy.
What's more concerning is the evaluation structure. The test set is a high-quality set painstakingly curated by hand, but it's one the authors assembled themselves. On top of that, the model used as the final scoring judge is the very same model used in the data-building stages — picking objects, composing edit requests, filtering for quality. It's a bit like a test-writer grading their own ability with a mock exam they wrote themselves. If only the kinds of edits that judge handles well make it into the data, and a model trained on that data is then graded by that same judge, there's room for scores to inflate. The reinforcement learning's reward judge was split off to a different model, but the independence of the judge that assigned the final scores remains a point to think about.
There's also a dependence on the base. VibeEdit is a structure that bolts a small adapter onto an existing editing AI, so it inherits a good deal of the underlying editor's skill and quirks. The novelty of the "instruct-by-marking" interface and the quality of the engine actually painting the picture underneath are separate axes.
Real-time behavior is an open question. Canvas editing fundamentally presumes an interactive tool where the user draws and immediately gets a result back. Whether the perceived speed from drawing a circle to getting a result feels pleasant in a real workflow is something you can only find out by using the tool directly.
ONTLE dispatches an AI project manager (PM) to each client project, managing the flow from intake of a request through analysis, development, and service exposure. If you had to pick one point in this flow where friction is most frequent, it's the deliverable-review stage.
The AI PM shows the client a web deliverable like a landing page or a card-news set and takes in revision requests. Clients usually write something like this: "Not the right-hand button on the top banner — make the small button next to it darker." When there are several buttons on screen, the PM has to ask back: "Which button do you mean?" That adds another round-trip to the intake loop, and sometimes the PM guesses and a deliverable with the wrong element changed gets sent back up. The very asymmetry this paper pointed out — "what to change is easy, which one is hard" — replays daily at ONTLE's intake window.
If the client could just circle that small button right on top of the deliverable preview and write "make it darker," then the moment the request is filed, at least the "which one" is narrowed down. ONTLE already has a deliverable viewer (/artifacts) that displays images, PDFs, and Excel files, so laying a mark layer over that preview is a natural extension.
And here this paper's most practical lesson applies directly — layer decoupling. When the PM passes the client's marked-up screenshot to the execution stage, the receiving side has to read those circles and notes as instructions, not mistake them for a picture to draw into the deliverable. Only by structurally separating the instruction layer from the original deliverable can "the marks asking for a fix" avoid leaking into "the fixed deliverable."
That said, a mark alone doesn't solve everything. Two things remain. One, what the mark eliminates is the round-trip spent explaining "which one" — not the ambiguity baked into the request itself, like "how dark is darker." Two, the circle the client drew on screen is, in the end, just a coordinate. Connecting that coordinate to the element that actually has to be fixed — that button's code or component — is a problem image editing never had, one ONTLE has to solve on its own. So canvas intake isn't a mere screen convenience; it's a change to the very data structure that holds a request.
If you map this paper's results onto your own service, here's what's worth checking by role.
- Automated data generation: an open-vocabulary object detector (Florence-2) pulls object candidates, overly small or large ones are filtered out and numbered, then a judge model (an image-and-language-understanding AI) picks objects that are easy to recognize and well-separated from their surroundings. A segmentation tool (SAM 3) builds an object mask from the selected box. Add and delete use an object-removal tool (ObjectClear) to make a pair, with the removed direction used for delete training and the reverse for add; swap uses an inpainting tool (FLUX.1 Fill); attribute change and move are generated by the base editor (Qwen-Image-Edit). The generated pairs are then evaluated again for quality by the same judge model and filtered. - Supervised learning (stage one): region-weighted supervised learning. A large weight is placed on the edit target and mark regions while full supervision is kept, and move edits weight a combination of the start- and end-position masks. The mask is slightly widened around marks and boundaries (dilation) to absorb rough marking, near-boundary effects, and downsampling. This supervision mask is used only in the training loss and is not given at inference time. Canvas marks aren't stored in advance but drawn on the fly during training (online rendering), varying stroke shape, thickness, and handwritten/printed typeface to expose multiple renderings of the same edit. - Reinforcement learning (stage two): rubric-based reward. The reward judge is a separate judge model, different from the one used for data generation and final scoring. Optimization uses DiffusionNFT, which doesn't store the full denoising trajectory. The reinforcement learning set is built from 3,520 conditions with large reward variance (five edit types × 704 each). - Model makeup: the image-and-language-understanding encoder (Qwen2.5-VL) and the pixel-generation parts (VAE, DiT) are all frozen, and only rank-128 LoRA adapters on the DiT's attention-projection and modulation layers are trained. The base model is Qwen-Image-Edit. - Resources / inference: both stages use 32 NVIDIA A100s. Reinforcement learning starts from the 12,000-step supervised checkpoint and is evaluated at 220 steps, with 12 rollouts per condition. Inference outputs one image per case; the baseline model uses 30 denoising steps (CFG 4) and the reinforcement-learning model uses 16 steps (CFG 1). - Computing the overall edit score: the pass rate of binary questions on edit success, surrounding preservation, and local quality is averaged per edit type and then averaged equally across the five types (%). This is viewed alongside the original-versus-output PSNR outside the edit region. - VibeEdit scores by edit type: add 80.3 · delete 85.4 · attribute 77.4 · swap 79.9 · move 76.4.
The paper has a public project page (https://zhaojingjing713.github.io/VibeEdit/). That said, within the scope excerpted for this article, we can't confirm whether public weights or a released tool exist, so we recommend checking directly on the project page and the paper link.
All of them are based on the arXiv public version and the project page in the references below. The scores in the body are the values the authors reported, and we recommend reading them with the caveat — as noted in "Limitations and Open Questions" — that those scores came on a test and scoring prompts the authors built themselves, with a separate external model as the judge.