Say you tell an AI that generates video in real time: "Paint the car on screen red." The color changes correctly, but the car has drifted to the wrong spot. Or a few seconds later you say, "Actually, undo that color," and the red stubbornly stays. The picture may look fine, but the job you asked for is wrong.
StreamEval is a benchmark (a standard for measuring performance) built around exactly this question. Is the ability to render things beautifully the same as the ability to do what you were told? The systems under test are "interactive streaming video generation" models. Rather than producing a clip in one shot and calling it done, these models keep generating video while accepting new user requests in real time, mid-stream.
Those requests fall into five tasks: talking with an avatar, directing an avatar to perform actions (performance), building a virtual space (world generation), editing video through spoken instructions (instruction editing), and editing to match a reference image (reference editing).
Until now, video generation evaluation has mostly asked "how convincing is one finished clip." VBench and EvalCrafter look at visual quality and stability; T2V-CompBench checks whether a scene matches its description. All of them score a single, already-completed video.
Streaming generation is different. While the video flows without stopping, requests like "turn your gaze to the left," "make this object red," or "undo that last edit" cut in at their own timings. Each request carries three things: when to apply it (timing), what to touch, and what to leave alone (preservation). Existing evaluations had no framework that specifies and measures these three per task. That is the gap StreamEval sets out to fill.
It is not that streaming-focused evaluations didn't exist. But the prior ones each concentrated on a single area, whereas StreamEval binds them under one session protocol and, per task, spells out "what to do and what to protect." It also covers a wider range of systems.
| Benchmark | What it measures | Systems covered |
|---|---|---|
| WBench / WBench-Editing | Continuous editing and preservation within a generated world | 20 |
| WorldMark | World-generation family | 10 |
| StreamAV-Bench | Interactive audio-video updates | 13 |
| LongV2VBench | Editing of long source streams | — |
| VideoFDB | Nonverbal audiovisual cues in conversational agents | — |
| StreamEval | The above tasks unified under one session protocol + explicit execution and preservation criteria | 37 |
Broad does not mean it covers every fine-grained capability, the authors are careful to note. For example, the "perceiving a user's video in real time" ability that VideoFDB measures is complementary to the directed dialogue and performance StreamEval measures, not a superset of it.
StreamEval frames each test as a "scripted session." Every session is given initial assets (a portrait of a person, a scene, a source video); on top of that, requests arrive one by one on a timetable, bundled with an answer key for "what to change and what to keep."
Look at the car again. One session of the editing task flows like this: turn the car in the flowing video red, hold that state for a while without touching the rest of the scene, then apply "undo the car's color." The crucial part is what happens when the undo comes in. The car's color should return to the original, but the car's position should be where it is in the video at that very moment—not where it was at the start—because the video has kept flowing the whole time. The paper defines this undo as "restoring the contemporaneous source state": reverting an earlier edit to the source as it stands at that moment. It is not "selective restoration," where you single out one particular edit to cancel while keeping others alive (that is explicitly listed as a future extension in §4.2). Changing the color (execution), keeping the rest of the scene (preservation), and reverting to the source at that moment (restoration) all live inside a single scene.
It also matters that the input is not handed over all at once up front. A scheduler releases requests one at a time against the session clock, and the source video being edited keeps advancing on the same clock.
Scoring looks at "a single request" from several angles. There are four groups, and under them ten metrics (all on a 0–100 scale). It's easiest to see which failure each metric catches by going back to the car.
Among these, what StreamEval drills into most is execution, preservation, and restoration. Restoration (reverting an earlier edit to the contemporaneous source state, as in "undo that color") is checked within Edit Success. The English labels are selected and applied to fit each task. For instance, Following is used for avatar performance and world generation, and Edit Success for the two editing tasks. Because each metric has a fixed "what it measures," results can be read split into execution versus fidelity.
The gap the title promises—between "looking good" and "doing what you asked"—shows up most sharply inside a single model. MuseTalk 1.5 scores Quality 95.34, Stability 97.11, and Identity 88.44, putting it in the top tier for appearance and stability. Yet A–V Sync drops to 38.09 and Dialogue to 54.34. It's a "talking face" that looks flawless, but the scores fall sharply once you move toward lining up mouth and sound and meeting the dialogue requirements.
LiveAvatar FULL is more extreme: Stability 100.00 with A–V Sync 0.00. But these two measure different things. Stability looks at how little the image jitters between frames; A–V Sync looks at whether mouth and sound match. A score for a stable image and a score for responding to instructions are not on the same axis.
Where the gap opens widest is "fulfilling the request." Below are the highest scores on each task (the best, not the average).
These three are top scores from different tasks and conditions, so they shouldn't be lumped together. Following is "the fraction of control events in which the requested manipulation actually happened to the specified target"; Edit Success is "the fraction of complete edits that pass all checks for scope, active state, deadline, duration, and protected content." Put plainly: even the best model at avatar performance did only about 37% of the directed actions as specified, and even the best model at reference editing completed only 15% of edits fully.
That said, this picture of "low execution, high fidelity" does not hold the same way across all tasks. In avatar, performance, and world generation, fidelity and stability far outrun execution—but instruction editing is the opposite. Vidu S2 Editing, the leader in instruction editing, scores Following 72.10 and Edit Success 57.20, high on execution, while Quality 22.99 and Stability 13.94 collapse on fidelity and stability (the paper also notes instruction editing's low fidelity and stability). In other words, the conclusion that "looking good" and "doing what you asked" diverge still holds, but which side lags depends on the task. The table below collects which model led on each task and how scores split by setup.
How to read this. Scores are aggregated as: average over 3 runs → average over cases → average over the five length buckets, with equal weight given to all five lengths (a safeguard against piling up points in short sessions). Also, the "number of models reported" for the editing tasks is not the full evaluation pool but a presentation subset of only those with Overall of 5 or higher. This selection was made after evaluation was complete.
Overall leaders by task
| Task | Leading model |
|---|---|
| Avatar dialogue / performance | Vidu S2 Avatar |
| World generation | MemFlow |
| Instruction editing / reference editing | Vidu S2 Editing |
Drop down to individual metrics and other names surface. The Quality winner in avatar performance is LiveTalk; the Preservation winner in instruction editing is JoyAI-Video-Edit. No single model takes every dimension.
The gap shows even within the leading models. In avatar dialogue, Vidu S2 Avatar scores Quality 96.78 and Identity 90.12, high on appearance, but Dialogue is 75.29. Its task Overall is 93.98, which is Q-Overall 88.98 plus a 5-point customization bonus (see the FAQ below on the bonus).
In the fixed-avatar setting, Identity is compared against the system's own first frame.
| Fixed avatar | Q-Overall | Identity |
|---|---|---|
| Tavus CVI Full | 86.65 | 89.89 |
| LiveAvatar FULL | 65.07 | 87.70 |
Edit Success is "the fraction of complete edits." For a single edit to count as a success, it has to pass four checks at once: ① did it select only the specified target (scope), ② did it preserve what should not be changed (protected content), ③ did it apply within the set deadline (deadline), and ④ did it hold that result through to the end (duration). An edit counts as one only if it clears all four, so missing even one means that edit does not count as a success.
This is what sets it apart from individual quality scores. Quality grades "how convincing" something is on a continuous 0–100 scale, but Edit Success is closer to a pass/fail of "did it get everything right or not." That's why you get cases where quality is high but the edit success rate sits well below it. The picture can look fine while one of the four conditions was missed.
That said, the overall score doesn't tell you why a low score is low. Whether it picked the wrong target, touched something it should have protected, or missed the deadline—you have to measure those separately to know. For instance, you could take the same edit request and layer conditions on one at a time: first just specify the target, then add a preservation condition ("leave this as is"), then a timing condition ("within a few seconds"), and watch which condition breaks the success rate. This isn't a result from the paper; it's the next question the results leave behind.
If you're only going to read the result numbers, you can skip this. For those who want to run it themselves, the construction and validation details are folded into an appendix.
**Scale.** It evaluated 37 models across 47 task configurations (a model that appears in multiple tasks counts once as a model but once per task as a configuration). Configuration counts per task are: avatar dialogue 15, avatar performance 7, world generation 17 (text 12 + image 5), instruction editing 5, reference editing 3. **Cases.** Each task has 60 cases, with each case spread evenly—12 apiece—across the five lengths of 15, 30, 60, 120, and 300 seconds. The 300 cases span 240 independent source families, and 60 text-dialogue derivatives (which replace a speech turn with same-identity text) reuse existing families. A full pass is 525 minutes of pure streaming generation and 1,347 scheduled online events. Each case and model is run independently 3 times. **Aggregation.** The task overall (Q-Overall) is the equal-weight average of the applicable metrics (four for world generation, five for the rest). The two avatar tasks carry a customization bonus, so the ranking Overall can rise as high as 105 (see FAQ). **Construction.** Session goals, timings, protected attributes, and draft expected outcomes are written by Qwen3.8-Max, while the "oracles"—spelling out what to establish, maintain, cancel, or restore at each checkpoint—are authored by humans. Portraits and visual references are synthesized with GPT-Image-2 and Qwen-Image (synthetic adults or stylized fictional characters, not real people), and speech is generated with Qwen3.5-Omni. Edit sources use licensed live-action and animation from Wikimedia Commons, NASA, Blender Studio, and so on, plus benchmark-generated scenes, and cover the full session without loops or speed changes. **Validation.** Synthetic speech must have a Qwen3-ASR transcription within a normalized word error rate of 0.25 against the script, and images are audited for sharpness and defects by Qwen3.8-Max. The automatic scoring judges are calibrated to a standard of at most 5% false positives, 10% false negatives, and 10% undecided per class, and for Identity the lower of the two judges' scores (Qwen3.8-Max-0902 and Qwen3-VL-Plus) is taken to prevent a person being swapped out within the same art style.
Start with the limitations the authors state. A–V Sync is measured by the correlation (≥0.3) between mouth opening and the speech envelope, but because it samples the source sparsely (500ms for dialogue, 250ms for performance), very fine lip-sync errors fall outside this metric's resolution. Also, items the judge couldn't be confident about ("undecided") create an interval with a lower bound of 0 and an upper bound of 100, and the table carries the lower bound. That means the more uncertain items there are, the more conservatively (lower) that model's score lands.
A few more points are worth flagging. First, the Qwen family handles the script and oracle drafts as well as the image audit. There is a structural opening for the preferences or blind spots of one model family to seep into the evaluation design (the authors try to suppress this through judge calibration). Second, the people are all synthetic and the edit sources are licensed live-action, animation, and generated scenes. A gap remains from the real-world setting where a user drops in their own face and their own video. Third, this is a leaderboard snapshot at a particular moment. In a field where models update fast, the structural conclusion—"fidelity and execution don't move together, and which side lags also changes by task"—lasts longer than the absolute scores.
ONTLE is a service that uses AI to help with the repetitive work of small businesses and solo founders who find it hard to hire more people. What it handles is documents, schedules, and budgets rather than video—but the evaluation structure of "apply instructions that arrive at staggered times while keeping what's already settled" transfers directly, because the session design (initial state + requests arriving on a timetable + a per-checkpoint answer key for "establish, maintain, cancel, restore") is the same whether it's video or office work. The automatic scoring criteria, however, have to be set anew for the work domain.
Here's one session worth putting into ONTLE's own QA.
The correct answer is clear. The only thing to revert in that last request is the 'budget cut' (budget restored to its value just before the cut), while the Plan A lock and the newly added channel B must be left as is. Translated StreamEval-style, you'd score Following (did it change only what was specified), Preservation (did it keep the settled items), and Edit Success (did the cancel, restore, and add all pass their conditions) at the session level. One thing to flag, though: this kind of 'selective restoration'—singling out one particular change to revert while keeping other changes (Plan A, channel B) alive—is not in the scope StreamEval currently evaluates; it's the area it lists as a future extension in §4.2. The restoration the current benchmark measures is reverting an earlier edit to the contemporaneous source state. So if ONTLE's QA aims at this selective restoration too, it's gone one step beyond the current benchmark.
What you're checking here is not whether the output looks good, but whether, across repeated instructions, it preserved the settled items and canceled only the specific change. Cancel and restore are also where people often slip. If ONTLE's QA doesn't stop at confirming a nicely formatted report but actually runs sessions like this, then for that one capability, at least, it can answer with numbers.
If you apply this benchmark's framework to your own service (conversational, agentic, or streaming products), here are items to check right away.
Only the two avatar tasks, dialogue and performance, have a "customization bonus." A system that lets the user supply the avatar they want gets +5, so the ranking Overall can go up to 105, while a system that only uses a vendor-fixed avatar gets 0. This bonus merely flags interface capability; it does not change the underlying Q-Overall score on the 0–100 scale. That's why Q-Overall and the ranking Overall should be read as distinct.
The leaderboard is at https://streameval.ai, and the paper excerpt also lists links to GitHub (github.com/StreamEval/StreamEval) and the HuggingFace dataset (huggingface.co/datasets/StreamEval/StreamEval). The team states it will release all cases including timelines, oracles, and media, and for third-party assets in the edit sources, per-asset licenses and provenance are provided with the data release.