Picture a movie. The protagonist is in the living room, walks off to the laundry room, and much later comes back to the living room. That returning living room has to be the same living room. The couch is where it was, the windows face the same way, the things on the table are still there. What's second nature to a human director turns out to be surprisingly hard for an AI that takes a script and generates the scene one piece at a time.
ATAS (Any Time, Any Space) solves this not by building a new video model, but by bolting two kinds of "memory" onto an existing video generator. One is scene memory, which holds the spatial layout of a room; the other is event memory, which holds what happened there. And rather than trusting a freshly generated scene outright, it writes only what passes verification into that memory. The authors report that across four video generators, shot-to-shot consistency metrics all improved — a shot being one slice of a long video — and that ATAS earned the top overall score with each generator.
Why is something easy for a person hard for an AI? Existing systems have mostly maintained continuity between shots in three ways: by passing along the last few frames, by handing over a text summary, or by retrieving and attaching a reference image. For shots that follow one right after another, this is enough.
The trouble is "non-contiguous" requests. When several unrelated scenes sit in between and you then ask to go back to that earlier room, or to bring that earlier event back, these cues are useless. The previous frames only hold the laundry room from a moment ago; they don't carry the living-room layout from three shots back on the same reference coordinates. The paper frames this as a problem of having no persistent world state — the shared record of scenes and events that survives shot changes. Maintaining that state takes two things. First, past observations have to be stored so they can be retrieved along both the spatial and the temporal axis. Second, you can't blindly accept the errors in what gets generated. If the visual and geometric errors of a generated scene flow straight into the state, one room that came out wrong will contaminate every return shot after it.
Three lines of work have gone at this problem. First, the camp that simply stretches long-video generation itself — looking at more context at once, or constraining camera and motion — never settles how to store observations from multiple clips and pull them back out. Second, agentic video systems — an AI that takes a script and decides, over multiple passes, what the next shot should be — bolt on script analysis, shot planning, and review, but what they carry between shots is usually "editing context" like prompts or reference images, with no shared coordinate frame and no event-time index. Third, the explicit-memory camp remembers space as point clouds or 3D representations, but has no step that decides whether to accept a generated observation as a future state. ATAS differs on two counts: it splits spatial memory from event memory, and it verifies an observation before committing it to memory.
Boil the design down to one sentence: ATAS separates video production into "the visible output" and "the evidence worth remembering." Memory splits into two kinds.
Scene memory stores the spatial structure of a room in a shared coordinate frame — laying every observation on the same reference axes. When the first shot of the living room comes in, that becomes the coordinate reference for the room, and every scene accepted afterward is aligned to that reference and merged into a static point cloud, which represents space as the coordinates of a great many points. Moving regions are dropped here, leaving only the fixed structure. When a later request wants to return to that living room, the stored coordinates are re-rendered from the requested position and camera angle and fed to the generator as a reference image. Think of it as a blueprint archive that remembers the skeleton of the house in 3D and can redraw it from any angle on demand.
Event memory keeps the moving scenes separate. It is indexed by which scene, what event, and when, and holds the per-frame evidence that registers the event in the scene coordinate frame (RGB-D, masks, dynamic point clouds) along with metadata like source and time. When a later request asks to bring back that earlier event over that time span, the index finds it immediately; if there's no matching record, it returns an empty result. The important part: this memory doesn't store a tidied-up conclusion like "someone moved this way" — it preserves the observations of the moment as per-frame evidence on the scene coordinate frame. That is different from keeping the whole source video.
Here's the full flow. Given a script, a "world-state compiler" breaks the long script into small shot requests sized to the length the generator can produce in one pass — recursively splitting a complex script into ever smaller units. Each request carries what to film, which scene it is, roughly where in the room and with what camera to view it, which time span, and, if it's an event, which event. These requests query the two memories to build context, and conditioned on that context the generator produces a short clip.
A generated clip does not go straight into memory. First it becomes a "candidate observation" and faces two separate judgments. One is video review — is this clip good enough for the final video? The other is registration — does the reconstructed result align properly to the scene coordinate frame? The state updates only when both pass and the source, scene scope, and prior state version all match. "Good enough for the final video" and "trustworthy enough to commit to memory" are different questions. A clip that works as video but fails coordinate alignment stays in the final video but is not used as a condition for later shots. This separation cuts off the path by which a misaligned observation leaks into the world state and poisons the next shot.
The evaluation is built from standard benchmarks, persistence tasks, and controlled analyses. For a reader, the fastest way in is to bundle it into three questions. Do we return to the same room? Do we reproduce the same event? And how much does it cost? The comparisons cover four generators (MiniMax-H3, Wan 2.7, HappyHorse 1.0, Wan 3.0) and five agents (I2V Extension, VGoT, Mora, MovieAgent, ReCA), and within each generator block everything is compared under the same conditions.
Measured on standard benchmarks (public evaluation tools like VBench, ST-Bench, and ViStory), ATAS led every reported overall score on MiniMax-H3, Wan 3.0, and HappyHorse 1.0. On Wan 3.0, for example, it posts an overall average of 0.734, ahead of the runner-up ReCA (0.648). On Wan 2.7 it narrowly trailed ReCA (0.854) on one VBench item (ATAS 0.846), but had the highest overall average (0.656 vs. 0.612). The same pattern showed up whether the generator was open-source or commercial. The authors read this gain as coming not from any one generator being good, but from the persistent state that runs across shots.
The method is this. The generated return scene is compared against the actual target image, using only the region visible in both frames. The metrics are PSNR_C and SSIM_C (image-similarity scores where higher is better) and LPIPS_C (a perceptual distance where lower is better). The comparison set includes not only agentic methods but also methods that redraw a scene in 3D (FlexWorld, ViewCrafter). The result: ATAS led the field at PSNR_C 15.968, SSIM_C 0.385, and LPIPS_C 0.284.
An AI that understands video and text together (a vision-language model) is given the original event frames, the event description and span, the requested camera, and the generated clip, and asked to score on five axes. The point isn't whether the same video was copied exactly, but whether that event comes back to life from that camera. That's why no synchronized target video is used.
| Axis | ATAS | Note |
|---|---|---|
| Event identity | 0.750 | Leading |
| Temporal order | 0.688 | Leading |
| Viewpoint realization | 0.650 | Leading |
| Character/object consistency | 0.917 | Tied with ReCA |
| Motion/state preservation | 0.688 | Narrowly behind ReCA's 0.700 |
The last row of this table marks the dividing line. The remaining difficulty is not in "finding" the event, but in "reproducing" the motion and state exactly. These are two different jobs. One is locating the right past evidence and handing it to the generator; the other is the generator actually drawing the motion as that evidence prescribes. ATAS owns the former. The latter rests on the generator's own capability — even with the right evidence in hand, if the generator can't faithfully draw the same motion, this score falls behind.
They vary the amount of evidence stored to measure memory cost, and measure runtime, retries, and throughput across six memory configurations.
| Metric | No memory | Persistent memory |
|---|---|---|
| Retries per shot | 1.00 | 0.54 |
| Delivery time including retries | 359 s | 339 s |
| Runtime per story | 51.9 min | 64.9 min |
| Throughput | 7.28 shots/hr | 6.21 shots/hr |
| Memory overhead per shot | — | 98 s |
In short, it's a trade. You pay the cost of building and verifying memory, and in return you cut the wasted re-generation that comes from a wrong state propagating downstream. Total compute itself goes up.
The effect of the verification gate has to be read in two parts. One is the immediate effect. The more memory accumulated, the lower the video-review rejection rate fell — from 52.6% (no memory) to 36.8% (full state) — steadily, and since the intermediate configurations all point the same way, this improvement isn't something that only appeared in the final setup. The other is the across-shots effect: it stops a once-misaligned observation from spreading into later shots and compounding into accumulated error. Up to here, these are confirmed results. That said, the authors read the fact that the score didn't jump much at the final step of adding all the components as meaning that verification preserves existing capability rather than adding new information. This part is the authors' interpretation, not a measurement.
| Component added | Change |
|---|---|
| Frame history | All three metrics up vs. no memory |
| Static scene evidence | LPIPS_C down 0.030 (improvement) |
| Scene update | C-Top10 up 0.041 |
| Event memory | Recapture score up sharply, spatial change small |
| Recursive compilation | Event score 0.508 → 0.711 |
The authors own up to some limits. On event recapture, exact reproduction of motion and state fell narrowly behind ReCA. In human evaluation, ATAS averaged first on all six criteria, but the spread across criteria was large.
| Criterion | Average |
|---|---|
| Script faithfulness | 4.40 |
| Character consistency | 4.22 |
| Visual appeal | 4.10 |
| Narrative coherence | 3.77 |
| Background consistency | 3.62 |
People clearly feel whether a shot is faithful to the script and whether characters carry through, but they feel that whole-environment consistency and the flow of the whole story are still less resolved. The authors also don't hide that total compute goes up on the efficiency front.
There are questions still unanswered. The approach leans on the premise that "the first observation sets the coordinate frame correctly" — and if the first registration comes in tilted, every alignment after it is bound to that reference. How the scene-memory design, which keeps only the fixed structure and drops the moving regions, behaves in stories where furniture gets moved or the lighting changes — where "what was believed static" shifts — hasn't been confirmed. That the comparison is limited to four generators, five agents, and a handful of 3D methods is also worth keeping in mind when gauging how well it generalizes to other settings.
This is a video paper, but what's useful to us — working on AI PM agents — isn't the video tech. It's the state-update rule.
Picture a scene. An AI project manager accumulates the state of a project across many days and many sessions. Yesterday's meeting settled on "payments reuse the existing merchant chain," but the summary the agent produces today slips in a sentence saying "payments will proceed via a new Toss PG application." Plausible, but wrong. In a typical design, that summary goes straight into project memory, and when prepping next week's meeting the contaminated state gets read back and reproduces the same error. This is exactly the propagation ATAS set out to stop in video.
Port ATAS's prescription over and it forks two ways. First, separate shipping the output from registering the state. The "payments = new application" error in the summary gets corrected at the report stage before it goes out. At the same time, for that decision to be registered as fact in the project's persistent state, it has to pass a separate check against the original meeting record, its source, and consistency with the prior state version. Fixing and shipping the report, and pinning its content into long-term memory as fact, are different judgments. Second, keep the state not as a "tidied conclusion" but as "sourced evidence, indexed by when, where, and what came out." Only then can you later retrace "when and in which meeting did that decision come up?" by channel (space) and date (time), and preserve the source of the decision.
It's also worth seeing that attaching a verification gate isn't free. As in ATAS, processing overhead gets added and total work can rise. In exchange, the rework caused by a wrong state spreading downstream goes down. Whether this trade is actually a net win in our work, though, is a question that stays open until you directly measure the rework reduced against the added cost.
If you pick just one thing to do first, start by checking whether your agent's output currently handles "delivery" and "memory commit" in the same single decision. If the two are fused, that's the first chokepoint where contamination spreads.
In the comparison, the methods within a block share the same generator, prompt, length budget, and evaluation protocol, and ATAS holds this interface fixed across the comparisons. The evidence that it's not a setup favoring one generator is that the same improvement pattern showed up on all four generators.
No. ATAS is a framework laid on top of a generator, so it doesn't retrain the generator. It only runs rules from the outside: break the script into shot requests, query memory for evidence and feed it in as a condition, and verify the generated clip to update memory.