Here's what output from video-generation models often looks like: the stage set is dressed to perfection, but the sign hung up as a prop is covered in illegible scribbles. A commercial as slick as a film, except the lettering on the product packaging is misspelled. VTR-Bench is a systematic benchmark that isolates and measures exactly this — "visual text rendering," the ability to draw legible, correct text inside a video.
The bottom line first: the measurements boil down to three things.
The scope: 300 prompts across 5 use-case scenarios, testing 11 recent models.
Evaluation of video-generation models has mostly looked at just one side. Is the image quality good, is it pleasing to look at, does it violate the laws of physics. That's broadly the axis existing benchmarks like VBench and EvalCrafter measure. But real-world video is full of text. Product descriptions in ads, figures in science videos, button labels on a screen. Get one character wrong and the information being conveyed changes.
The catch is that no matter how realistic a video looks, there's no guarantee the text inside it is correct. An example at the top of the paper shows this gap sharply. A roughly 10-second video scored a Video Score of 0.90 on a checklist of scene and motion requirements — yet the same video had a WER of 0.552. That means it contained spelling errors, repeated and nonsensical characters, and phrases that were simply unreadable.
It's not that no benchmark had ever evaluated text, but the limitations were clear. T2VTextBench relies entirely on human evaluation, so re-measuring a model repeatedly keeps costing human labor. EvalCrafter and AVGen-Bench do touch on text, but they focus on short strings and can't properly measure long sentences. Meanwhile, there have also been attempts to overhaul the models themselves so they draw text well. HunyuanVideo 1.5 encoded glyph-shape information into the model, and other work fine-tuned models on synthetic, text-heavy video.
VTR-Bench's place is different from all of these. It is not a technique for fixing models — it's a ruler. It chose to measure scene text, including long sentences, automatically, with no human in the loop, in a way that keeps working as you add more models. And it doesn't stop at measuring: it also proposes a path to reduce text errors by changing only how a model generates, leaving the model's weights untouched. "Measure precisely, improve without training" — that's the ground VTR-Bench stands on.
The results table sends a clear message. Writing text is a shared difficulty for all of today's models.
The overall winner was the commercial model Wan3.0. It led on text accuracy across all five scenarios (ads, science, UI, culture, everyday) with a WER of 0.250. What's interesting is second place. It wasn't a closed commercial model — the open-source Minimax H3 (0.447) beat the commercial Seedance2.5 (0.641) to claim second overall. A public model wrote text more accurately than several commercial ones. Conversely, a good number of models had WERs pinned almost to the ceiling (around 1). Wan2.2-5B and LTX-2.3 both came in at 0.996, with their error scores glued nearly to the upper bound of 1.
The second finding proves why this benchmark exists: a high video score is no guarantee the text is right. The sharpest contrast is between Kling v3.0 and Seedance2.5. Both have a Video Score of about 0.79 — nearly identical. Yet their WERs are 0.979 for Kling v3.0 and 0.641 for Seedance2.5. They satisfy scene and motion requirements equally, yet their text accuracy diverges sharply. Fold these two abilities into a single score and this gap gets hidden entirely.
The failure analysis is practical too. The authors sorted 13,213 text targets into three states: couldn't find a place to put the text (the carrier), found the place but couldn't read the text, and transcribed. Across all models, the "found the place but couldn't read the text" rate was 23.31% — far higher than the 9.34% that "couldn't find the place at all." That's a signal that models stumble more on drawing legible characters onto a found surface than on finding a surface in the first place. And being readable doesn't mean being correct. Kling v3.0 had a text-readable rate as high as 79.62%, yet its WER was 0.979. "Readable" and "correct" are different problems.
The effect of resolution and length is also worth noting. Comparing Minimax H3 and LTX-2.3 across four combinations of high/low resolution × 10-second/5-second length, Minimax H3's WER dropped at both lengths when resolution was raised. Given more pixels for the text, its existing ability came through more. LTX-2.3, by contrast, stayed pinned near a WER of 1 in all four combinations. The authors' reading is persuasive: the gains from changing settings only materialize when the model has text-writing ability to begin with. Without it, raising resolution just makes the wrong characters crisper.
Finally, the effect of improving "only how generation is done." Three approaches were compared on Minimax H3: plain generation (Original), feeding only a single first image and having the model build the video from it (image-to-video, I2V), and the full framework (Agentic). The full framework cut WER from 0.4468 to 0.3015 — a 32.5% reduction — and raised the Video Score from 0.7562 to 0.8315. Both metrics improved together across all five scenarios. Feeding just the first image (I2V) also beat plain generation, but the full framework cut WER a further 9.4% beyond I2V. That means supplying only the first frame isn't enough; a process of taking feedback iteratively and tuning along the way added value.
One idea runs through VTR-Bench's design: separation. It measures, on two distinct axes, whether a video realizes the requested scene and motion well, and whether the text inside it is accurate. Fold them into one and "looks convincing" swallows "text is correct."
The first axis is the Video Score. Each prompt comes with 20 prepared questions, and a VLM (a model that watches video and answers in words) scores the video with yes/no answers. The questions cover five dimensions: scene attributes, motion compliance, spatial relations, entity presence, and temporal consistency. The Video Score is the fraction of requirements satisfied.
The second axis, the core of this paper, is text evaluation. The VLM picks the moment in the video where the surface bearing the text — the "carrier," be it a sign, a package, or a screen — is seen most clearly, and transcribes the text. It does this with the ground truth hidden, writing only what it sees: spelling errors kept as-is, unreadable parts left blank. That transcription is then compared against the ground truth with WER.
There's one more device here: bounded WER. To keep a model from endlessly spewing junk characters and sending the error score soaring without limit, the score is capped relative to the ground-truth length. That's why WER tops out around 1.
The standout step in building the dataset is drawing it out first and filtering. Rather than writing prompts in text and calling it done, the authors render the scene in advance with an image-generation model, then use a VLM to verify and fix: "can this text and this sign actually fit together into one scene?" If it fails three times in a row, the prompt is rewritten from scratch. It's a smart procedure that screens out, up front, the mistake of testing a model with an inherently impossible task.
The framework proposed for improvement rests on the same idea. A conducting role called the "Director" first receives several first-frame candidates, inspects them for text accuracy, legibility, and so on, and picks — or fixes — the best one. The video is then generated conditioned on this polished first frame. The point is that quality is raised purely through control at generation time, without ever touching the model's weights.
The experimental conditions are controlled. The targets are 6 recent open-source models and 5 closed commercial ones, 11 in all. Videos were standardized to 10 seconds at 24FPS, and commercial models were fixed at 720p resolution. The dataset's 300 prompts are evenly distributed, 60 per scenario across 5 scenarios, covering 25 sub-scenes beneath them. They mix short labels and long paragraphs (1–275 tokens per block, median 23), designed to test both the ability to draw multiple strings at once and the ability to reproduce long sentences accurately.
One thing to be clear about. This benchmark measures whether text "matches the ground truth" — it is not concerned with whether the font is pretty or natural. Also, because transcription is based on the "clearest moment," whether text holds steady throughout the video is handled not by WER but by a separate item.
Start with the limitation the authors acknowledge outright. They nail down, via the LTX-2.3 case, that setting adjustments alone don't solve text errors. Their stance is that a model's fundamental text-writing ability is central, and that resolution and length are supporting levers that amplify gains only when that ability is present. An honest line to draw.
A more structural limitation is that the evaluation leans on a VLM. Both text transcription and video scoring are performed by the same evaluation model. The authors cross-checked this against human evaluation and against results from using a different evaluator (in the appendix). Even so, it isn't easy to fully separate "text the evaluator couldn't read" from "text the model couldn't draw." The decision to have you look at transcription rate and WER together reads as a device for handling that boundary.
One more: the effect of the generation-method improvement was confirmed on a single model, Minimax H3. Whether a model with weak baseline ability — like LTX-2.3, whose WER was pinned to the ceiling — would see the same magnitude of improvement is hard to assert from this experiment alone. As the resolution analysis suggests, without baseline ability, control at generation time may also have limited effect. Just how far above some threshold a model must be for this method to work, and whether it's universal, needs broader validation.
Picture an AI PM producing a deliverable for a client. A summary report studded with figures; a marketing card-news piece introducing a product. When the layout is clean and every section is filled, so it looks finished, a human reviewer tends to lean on that sense of completeness and skip checking the content. In that gap, the report goes out to the client with a wrong figure in a chart, or a typo in the product name.
This doesn't mean it's the same problem as text errors in video. What's alike is the structure of the trap: when the surface looks convincingly complete, you skip verifying the accuracy inside. The very gap where Kling v3.0 scored a Video Score of 0.79 while posting a WER of 0.979 can open up just the same way in deliverable review.
So ONTLE's review pipeline has to split the two evaluations structurally. One is "form and completeness" (are all requested sections present, is the tone right), the other is "content accuracy" (do the product-name spelling, the price figures, the quoted numbers match the source). The latter can be automated the way VTR-Bench extracts only the text on the carrier and checks it against ground truth — by pulling only the "text to be verified" (numbers, proper nouns, quotations) from the deliverable and collating it character-by-character against the source data. Merge the two scores into one lump and issue only a "pass/hold," and content errors go out hidden behind a sense of formal completeness.
The improvement loop fits a PM's way of working too. Just as the paper cut WER by 32.5% through iterative feedback alone without touching the weights, deliverables improve when run not as "generate once and done" but as a loop of draft → content verification → targeted editing of only the wrong parts → selection. In particular, fixing only the erroneous text rather than regenerating the whole thing cuts the waste of producing fresh errors every time you regenerate. But heed the lesson of the resolution analysis as well: no amount of looping helps a generator with no baseline ability, and the verification loop is only a lever on top of a model that produces usable drafts.
No. WER is not a "fraction correct" — it's an error score computed by summing the token-level edit distance of insertions, deletions, and substitutions, normalizing by the ground-truth length, and counting each video up to a maximum of 1. So you can't convert 1−WER into an "accuracy rate." 0.250 means "a normalized edit-error score of 0.250," and the lower the value, the more accurately the text was written. A value near 1, like 0.996, means the error score is glued almost to the upper bound of 1.
No. WER only measures how well the text matches the ground truth. Aesthetic quality — whether the font is natural or pleasing — is not within this metric's scope. And because transcription is based on the "clearest moment," whether the text holds steady without flickering across the entire video (temporal stability) is handled separately from WER.