Researchers at Arena and UCLA pushed a single image model to the top of the open-source field on the Arena leaderboard (snapshot of 2026-09-04). The secret wasn't a bigger model or more data. It was abandoning the habit of measuring quality with one score, and instead combining several rewards of different temperaments so that they don't fight each other, tuned to the model.
Why doesn't one score work? When you try to pin down the images people like with a single score, that score goes up while actual quality stalls and then buckles. Chasing the number while ignoring real quality is what's called "reward hacking."
This isn't really a problem unique to image AI. Every attempt to measure the quality of an AI's output with a single number runs into the same dilemma. Our ONTLE team wrestles with this daily, managing the quality of what our AI agents produce. So we dug all the way into this "reward composition recipe."
A model that draws a picture when you feed it text is called a text-to-image (T2I) model. Reinforcement learning is a way to train such a model one more time so its output fits human taste. You reward good results with a score and iterate, so the model reshapes itself toward higher scores.
The problem is that in this training, the reward is everything. Even a slightly off reward drags the model not toward a better place but toward a "place that scores well." One figure in the paper (Fig 5) nails this down. When a model called FLUX.2-dev was optimized against a single reward, the measured reward value kept climbing while actual quality quickly plateaued and then got worse. It's the same as telling a student to raise only their test score: they pick up tricks instead of skill.
The rewards used so far learn "what people generally like" from millions of human comparisons. They're good at capturing aesthetics, composition, and overall impression broadly, but they're blind to attributes that only surface occasionally — whether every requested object made it in, whether a stated constraint was honored. A single broad eye lets these details slip right through.
Research on image-AI quality has split into two streams. One is "how to teach" (the algorithm); the other is "what makes a good image" (reward design). This paper stands on the second. It borrows an existing teaching engine and carves its path by redefining the very standard of "good" — from a single score to a multi-dimensional composition.
The fix is simple. If one eye can't see it, use several. Take one request as an example: "A quiet back alley on a snowy night, lit by neon signs, in a cartoon style. No people." The researchers look at this picture through four rewards, each in a different way.
| Reward | In a word | What it sees in this example |
|---|---|---|
| Preference | Overall impression | Is this alley picture appealing to a human eye? |
| Fidelity | Did what was asked make it in? | Are the snow, night, neon, back alley, and cartoon style actually there? |
| Constraint | Did something un-asked-for sneak in? | Was "no people" honored? |
| Anti-hacking | Is it exploiting loopholes? | Any tricks like broken text or excessive photorealism? |
The preference reward looks at overall impression. It uses the classic comparison method of pitting two images against each other and converting into a score which side is more likely to win. The researchers trained their own preference model on the millions of human votes accumulated on Arena[^1], and they also left room for the public model PickScore to drop straight into the same slot. What matters isn't "which model" but the role: assigning a score to the side people would prefer overall.
The fidelity reward is a checklist. First an LLM — an AI that answers questions posed in words — breaks the written prompt into a list of yes/no questions. Then an AI that answers by looking at the image (a vision-language model, VLM) goes down that list item by item and scores the proportion of "yes." It's the "is what should be there, there?" eye, confirming that the requested snow, neon, and back alley actually made it in.
The constraint reward is its mirror image. Using the same checklist framework, it asks whether un-asked-for clutter crept in. It covers not only explicit prohibitions like "no people" but also implicit constraints that weren't written out in words, like "keep it minimal."
The anti-hacking reward is a detector that catches whether the model is exploiting loopholes in the rewards. It splits in two. The part that can tell from a single image catches tricks like un-requested broken text popping up; the part that only becomes visible when placed side by side with the output of the untouched original model catches the tendency to quietly drift toward photographic realism when a cartoon style was requested. The latter needs something to compare against for "too much" to show.
Four of them might look like over-engineering. The researchers' qualitative comparison (Fig 1) shows improvement cases from Stage 1, before anti-hacking is turned on — that is, when preference, fidelity, and constraint are meshed together. That's where it becomes clear why any single reward falls short. Use only the preference reward and it drifts into plausible-looking decoration while ignoring the request. Use only the fidelity reward and details get smeared, with the requested style and background dropping out. The case where un-requested decoration simply remains comes not from fidelity alone but from Arena-T2I-Hard, which combines preference and fidelity — and that's exactly why the constraint reward is needed. But apply the constraint to any and every request and it flattens detail even where creativity was permitted. So you mesh preference, fidelity, and constraint together, but the constraint has to pick its moments (gating) to honor the request with no visible loss of quality (Fig 1). The need for the fourth reward, anti-hacking, comes not from Fig 1 but from the reward-hacking phenomenon we saw earlier (Fig 5). That's the answer to "why a composition."
Gathering the four ingredients isn't the end. Just adding them up makes the rewards interfere with each other and optimization wobbles. The researchers put three layers of safeguards in place so the rewards create synergy instead of fighting.
Normalization. Each reward has its own native numeric range. Add them as-is and one reward with a large range hogs the optimization. So they normalize each axis separately, per batch — putting the number scales on the same basis — and place them on the same scale. The trick is to align per-axis first, not after combining.
Gating, that is, turning on and off to fit the situation. Not every reward fits every request. Apply the constraint reward to an open-ended request like "a country road in an Impressionist style," and legitimate creation — adding one more flower — gets penalized. So they sort the request's intent into three buckets — spec-locked (strict), neutral, and creative-latitude (open) — and the constraint reward only turns on for strict. Anti-hacking works on the same principle. Realistic rendering is normal for a photo request but a failure for a cartoon request, so it only activates when it conflicts with the request's intent. Whether this intent classification is trustworthy was checked by having two humans label blind and measuring agreement with the classifier.
Veto. The anti-hacking signal isn't used as a bonus that adds to the score — it's used as a veto. When a trick is caught, it merely confiscates the preference points that picture earned; it does not reward the absence of a trick. Why must it go in only one direction? The moment you reward "no trickery," the model takes maxing out that reward — learning to fool the detector into "looking like it didn't" — as a new puzzle to solve. The veto adds no new objective to the reward and only suppresses the bad behavior itself, keeping the model from wandering off course.
The last two touches are surprisingly anticlimactic. When combining the rewards that have passed through the three safeguards, the researchers give every axis the same weight. No separate tuning. And it still worked consistently across two kinds of preference models and both base models. This means the composition recipe isn't a finicky formula that only clicks under specific conditions, but something closer to an "all-purpose seasoning" that works broadly across models — at least within the range the paper tested.
And then the "weight soup." They take the two outputs from training with anti-hacking on and off and average them 1:1 into one. What gets merged here isn't the whole model but the LoRA. LoRA is an approach that leaves the original weights alone and trains only a small add-on module. With no further training, it blends in just the strengths learned under the different configurations.
They didn't build the optimization engine from scratch. They took an existing algorithm for tuning diffusion models with reinforcement learning (DiffusionNFT) as-is. The paper's contribution lies not in "which algorithm to push with" but in "what to watch while pushing," and "how to bind those several things into one."
The core of the validation is whether this composition design carries across different models, and how they measured that improvement credibly.
There are two base models: FLUX.2-dev (32 billion parameters) and Ideogram 4 (9.3 billion parameters). They differ considerably in size and in internal architecture. Ideogram 4, in its base state, is already the No. 1 open-source model on the leaderboard. For both models, the original weights were frozen and only a LoRA was trained. If this composition design works on two models with different architectures, that's evidence the design isn't a trick fitted to one model. That said, the reward configuration layered onto the two models wasn't identical.
The prompts for training were drawn from Arena's real user distribution, but filtered aggressively, following the observation that reinforcement learning is especially sensitive to data quality. Edit requests, thin one-line prompts, ones dominated by rare proper nouns, and non-English were trimmed away, leaving 10,000 for training and 1,000 for validation.
Evaluation is on two levels, and they must not be read mixed together. One is the win rate assigned by an automatic grader. On the 1,000 validation items, they pit the model being trained against the original model and count which side wins more often, running it through two different graders to confirm the result doesn't lean on any single grader. The other is external validation: Elo, the votes that real users cast blind on the Arena leaderboard. Elo is a relative-skill score from chess — higher means people chose it more often.
These two carry different weight. The automatic grader's win rate is "improvement as the grader feels it," not the human's final judgment. So the headline numbers come from the human votes. FLUX.2-dev with this recipe fully applied scored 69 Elo points higher than the original model. For Ideogram-4, already No. 1 among open source, they used a configuration that added, to the preference, fidelity, and constraint rewards, an OCR reward that checks whether text was rendered correctly — but did not layer on the anti-hacking reward or the weight soup. That refined Ideogram-4 scored 1223.5, surpassing every open-source model on the leaderboard.
Let's be precise about scope. The same leaderboard also carries powerful closed models like nano-banana-2, seedream-5.0-pro, nano-banana-pro, and Qwen-Image-2.0-Pro. The Ideogram-4 result leading is strictly within the open-source range. Reproducibility was taken care of, too. The researchers released a 1,000-item subset that revives part of the full training effect, opening the door for others to reproduce this recipe lightly.
The researchers call out the limits separately. Here are the three dependencies that read clearly from the structure.
First, it's all inside one ecosystem. The preference reward was trained on Arena votes, the headline evaluation happens on the Arena leaderboard, and the authors' affiliation is Arena too. The researchers were aware of this and also used public PickScore and swapped graders to check whether the result carries over, but since the demonstrated gains come mostly on top of a single taste distribution, transfer to other tastes is an open question.
Second, grader dependence. Fidelity, constraint, and anti-hacking all sit on top of answers from AI graders (the LLM and VLM seen earlier). A reward's quality can't exceed the grader's quality. When the grader gets a particular question wrong, the reward is distorted to the same degree.
Third, the detectors only target "already-known" failures. They're defenses against observed tricks like broken text or excessive realism, so when optimization finds a new loophole, a new detector is needed. The veto does keep the detector itself from becoming the target of exploitation, but it can't preemptively block tricks that nobody knows about yet.
Back to the opening question. What would this design — the one that worked on an image model — look like transplanted into an AI-agent product like our ONTLE? First, let me nail something down: what follows is not a validated result but an applied hypothesis. There's no guarantee yet that a design that worked on an image model produces the same effect in product operations. Still, the design's principle can be borrowed as-is.
Let's follow one scene all the way through. A customer commissioned a product detail page: "Make it modern, and leave out the exaggerated ad copy." ONTLE's AI PM receives and analyzes the request, drafts a first version, and then — right before sending it to the customer — makes an acceptance judgment on its own.
The easiest implementation is to ask the judging model "Is this output good? Rate it 1–10" and tighten the pipeline so that score goes up. But this is exactly the trap Fig 5 warned about. Target a single score and the score climbs while the customer's felt quality can buckle at some point — because the PM will first pick up the knack for making "outputs that score well."
Transplant the composition structure onto this acceptance judgment, and what was a single number turns into a multi-dimensional report. For this detail page, it goes like this:
This customer request, too, mixes a spec-locked part ("no exaggerated copy") with a part open to creative latitude ("make it modern"). So gating fits naturally. Turn on "no clutter" only for the closed request, and don't penalize the designer's latitude on the open part. And if trick-detection is wired as confiscation (veto) rather than a bonus, the path of inflating the score by ticking form alone is blocked.
The more composite the goal, the more we can expect the team to move toward real skill rather than the knack for scoring points. It's a hypothesis, no more. That's why we read this paper all the way through with an agent product's eyes.
No. It freezes the base model's original weights and trains only a small add-on module (LoRA). The final "weight soup" step also just averages two already-trained outputs, so there's no additional training. The researchers' release of a 1,000-item subset is meant precisely so you can reproduce it without a heavy lift.
The paper reports giving every axis the same weight (no tuning) and that it worked consistently across several models. Whether more refined, situation-specific tuning yields additional gains is outside the range the paper tested. What's confirmed goes only as far as "equal weighting is a stable starting point."
The range this paper demonstrates is open-source models. It makes no claim about how it stacks up against the stronger closed top models, so whether it can beat them remains an open question.
[^1]: The in-house preference model is built by placing a "small output head that emits a single preference score" on top of a state-of-the-art large understanding model. It was trained on about 5 million vote pairs from the Arena leaderboard (2025-04-16 to 2026-06-07, with careless votes and low-quality prompts excluded).