RULER is a paper about the task of "taking a written instruction and producing SVG (vector graphics) code." It centers on a single question: when there is no one correct image to aim for, how do you actually train the model well? RULER's answer is to automatically generate a six-criterion scoring rubric for each instruction, and to use the score from that rubric as the reward for reinforcement learning. This approach clearly outperformed earlier methods that relied on off-the-shelf metrics. Exactly how much, and in what way, we'll see in the numbers later.
First, let's be clear about what SVG is. The JPG and PNG files we use every day build an image by packing dots (pixels) densely onto a grid. SVG, by contrast, draws using commands (code) like "draw a line from here to there, and fill this circle with blue." Because of that, it never gets blurry no matter how far you zoom in, which is why it's so common for icons and illustrations.
The trouble starts when you try to generate that SVG code from a natural-language instruction like "draw me a cat icon." This is an open-ended task — one with no single absolute right answer. A math problem has exactly one correct answer, but there could be hundreds of valid cat icons. And because there's no ground truth, two things get stuck at once. One is the job of judging whether the result is any good; the other is deciding the reward signal to hand out when improving the model with reinforcement learning (the score that tells it which outputs to prefer).
Earlier methods used scalar metrics like CLIP score or an aesthetic score — in other words, quality boiled down to a single number. But these metrics were calibrated against real photographs, so they don't fit stylized vector graphics well. It's like measuring a cartoon with a ruler made for photos. Worse, feeding such a metric straight into RL as the reward invites reward hacking. Reward hacking is when the model, instead of genuinely drawing better, finds shortcuts that merely push that score up. Think of a student who games the gaps in an exam to raise their grade without studying.
RULER's intuition is exactly the scoring rubric of a school performance assessment. If a teacher says "I'll just grade how well it's written with one number," the standard is vague. But if they break it into items — "Is the topic clear (2 pts)? Is the structure good (2 pts)? Is the spelling correct (1 pt)?" — the grading is far fairer and matches human judgment much better.
The paper confirmed this experimentally first. When a vision-language judge (a VLM — an AI that understands images and text together) was given a multi-axis rubric and asked to grade, it agreed with human judgment far more closely than a single-number metric did. And that held both when ranking different outputs against each other and when telling good from bad within the same instruction.
Built on top of that finding is RULER (Instance-aware Rubric Rewards for Reinforcement Learning). Let's walk through how it works using a made-up example for clarity: "draw me a smiling yellow sun icon."
The biggest practical advantage is that none of this requires a "correct" image or data that humans painstakingly graded. Because the scoring sheet is built purely from the instruction text, there's no need to pair up reference SVGs in advance, and no need for humans to attach preference labels. The entire cost of collecting ground-truth data drops out, sharply reducing the burden of building a dataset.
Now for the numbers. On two benchmarks (standard test sets used to measure model performance), the evaluation scores rose as follows.
| Benchmark | Before | After RULER |
|---|---|---|
| MMSVG-Illustration | 0.432 | 0.693 |
| MMSVG-Icon | 0.395 | 0.683 |
One thing worth flagging here. "Before" is the performance of the base model prior to training, and the scores in this table are measured separately from the reward used during training. RULER's training reward is given via a per-instruction rubric built differently for each prompt, but the performance evaluation follows an independent judging procedure that measures every output against one shared rubric. In other words, the ruler used to teach and the ruler used to grade are different. Scores closer to 1 are better, and both benchmarks jumped sharply. On top of that, RULER outperformed models that specialize exclusively in SVG generation, and even reached performance on par with the far larger DeepSeek-V3. A lighter method went head to head with a giant model and held its own.
So what produced this performance? The paper pins down the answer with an ablation (an experiment that removes components one at a time to see which was decisive). The real driver of the gains wasn't the RL technique itself but the rubric design. In other words, the key was what was graded, by which items, and how it was broken down.
To sum up: even in creative, open-ended generation tasks with no single correct answer, if you clearly break down "what counts as doing well" into items, you can build a reliable reward signal.
Of course, this approach has clear limitations and open questions too.
RULER's intuition sits right up against the core question of our own work: how do you evaluate something so that an AI can get good at a task with no right answer? We always have to avoid the error of measuring quality with a single metric that has nothing to do with the output's actual nature. How well a request was addressed, factual accuracy, tone, and actionability are different axes, and lumping them into one score buries the very flaws that matter most. RULER demonstrated empirically that breaking these axes apart and grading each one matches human judgment better. It also shows that even for outputs where you can't attach a correct answer in advance, you can extract evaluation items straight from the request text — and it comes with a warning: if you push performance toward a single metric, you get shortcuts (reward hacking) that only inflate that metric.
Let's make it concrete. Say an AI PM has drafted meeting notes from a client call. We apply the same evaluation procedure directly. (1) From the instruction and context of that meeting, we build an item sheet: "Are all the decisions captured? Do statements and facts line up? Is the tone appropriate to send to the client? Are the next steps written so they're actionable?" (2) We grade the draft item by item. (3) If only "actionability" comes out unusually low, we single out that item and send it back for a rewrite with feedback like "specify the owner and the deadline." That's a signal pointing at where and why it falls short, not a blanket "write it again." But to hand all of this grading over to an AI judge, there has to be a prior step that confirms the judge actually matches human judgment. That's precisely why it matters that RULER verified this correlation before the main training.
The scope RULER actually validated is two SVG-generation benchmarks. That said, the underlying principle — evaluating and improving an answerless task with an item-by-item scoring sheet — is worth trying on other tasks that have no single correct answer, like meeting notes or draft replies. The catch is that when the task changes, you have to re-validate accordingly; that, too, is one of the open questions the paper leaves behind.
When all you have to do is push one number up, shortcuts that attack just that number crop up easily. Conversely, when you look at several items separately and assign weights, forcing up a single item still gets you docked on the others — so making the output uniformly good becomes the winning strategy.