You shoot an enemy down, turn around, and walk away. The room you just passed through, the item dropped inside it, the half-open door — all of that is now off-screen. Come back a few minutes later: will any of it still be the way you left it?
This is exactly where the approach of drawing game frames with a video generation model most often falls apart. Whatever leaves the screen survives only inside the model's "memory" — and that memory fades the longer time goes on.
Matrix-Game 4.0 commits to a single principle. It does not entrust the current state of the world to the model's memory; code holds onto it separately. That lets it run a world that plays at 480p and 60FPS on a single RTX 5090 graphics card, with just 112ms of latency between a keypress and its appearance on screen.
Building controllable video with a generative model is not new as a pipeline. The Genie line of "interactive world models" — models that react to player input and produce the next frame in real time — has already shown it can be done. The trouble surfaces the moment the goal shifts from "take a quick look around" to "a long game with living rules." If you try to sustain a world on video generation alone, three things go wrong.
It forgets state. The state of the world lives only implicitly, inside the generated pixels and the model's context. Regions outside the field of view are not stored anywhere. So the room you return to has changed from the one you first saw.
Error accumulates. The model looks at the previous frame to produce the next one (this is called autoregression). With no external reference to correct it, small errors snowball frame by frame and the scene gradually collapses. Even if you bolt on an external 3D tool to estimate the camera pose, the error in that estimate accumulates and you hit the same wall in the end.
Rules are hard to enforce. Interaction stays limited to movement and simple environmental reactions. Judging a hit versus a miss, keeping the consequences of an action, setting goals for the player to chase — the rules that make a game a game are hard to enforce through video generation alone. On top of that, past frames just pile up without being organized into spatial information, so memory cost balloons the longer a session runs.
In the end, drawing a frame that looks like an explosion and computing the damage that explosion leaves behind and writing it into the world are two different jobs. A diffusion model (a generative model that learns by restoring an image from noise) only learns the statistical correlation of "what tends to appear alongside what"; it does not understand physics or causality. So the paper treats today's video generators as closer to a renderer that draws what it is told than to a model that understands the world.[^1]
Start with how this differs from prior approaches. In the Genie line, video generation also shoulders the world state — which is exactly where the weaknesses above point. Matrix-Game 4.0 goes the other way. It stands on the side where a video model renders a world defined in code, then combines that with real-time autoregressive generation and an AI agent that writes code directly. The aim is to move from "controllable video" to "an executable world."
To see how the pieces actually mesh, follow a single scene: the player shoots an oil drum on the floor. The work splits three ways. The engine adjudicates, the agent prepares the code and the descriptions, and the model produces the picture.
Engine — adjudication and memory. A lightweight game engine holds the world's state and rules in code. Whether the bullet hit the drum, and if so whether an explosion follows and how much health the surroundings lose — the engine decides all of that. And the engine does not draw a finished picture. The drum, the walls, the characters are left as plain shapes. The key point: the state of the drum that vanished in the blast and the scorched floor is something the engine keeps holding onto, even after the player turns away and sends it off-screen.
Agent — the one that prepares ahead. Between the engine and the model sits a multimodal coding agent ("multimodal" means it handles text and images together). What matters is that this agent does not step in on every frame; it works before play starts, or at the moment a rule changes.
It does two things. First, when the player asks for a new rule in words, it turns that into executable game code and adds it to the engine. A single line like "make monsters swarm in from all sides" demands several decisions the moment you put it into code — where they appear, how they move and attack, what decides win or loss. Whatever the player leaves unspecified, the agent fills in to fit the context of the existing scene. Open a door for which nothing has yet been defined, and the agent builds "a warehouse with a locked steel box in the corner," registers it with the engine, and from that moment on the box's state and whether it is open are managed by the engine.
Second, for every object that will appear in a scene, it attaches a description of what it looks like ahead of time. The idea is to tag a bland box with meaning — this material, this color. That description becomes raw material later, when the picture is drawn.
Model — the picture. The video generation model receives the plain-shape frame the engine hands over. The paper calls this shape frame a low-poly proxy — a skeletal frame that keeps only basic geometry with no texture. The model follows the structure and motion of that skeleton while dressing it in material, lighting, and mood. But following is not the same as guaranteeing: the official model card notes that over long sessions, under occlusion, or during fast camera moves, the appearance can drift and the geometry can break, so the generated video does not promise a faithful reproduction of the scene. The distinction matters — the engine's state stays exact in code; the picture the model draws from it carries no such guarantee. It continues the video conditioned on the current input and the frames up to just before.
During play, the real-time loop turns like this: player input → the engine updates state → shape frame and control signals → video rendering → visual feedback to the player. Note that the agent is not in this loop. The agent is not summoned to describe the scene every time a drum blows up. The engine makes the explosion call, and the "description" that conveys the result to the picture is filled in automatically from the engine's state values, as we will see. The code the agent wrote and the object descriptions it attached in advance are already prepared before this loop ever runs.
Splitting things this way separates the hard job of keeping the world consistent from the job where real-time speed is on the line. The engine only has to run a lightweight simulation, so the speed burden can be concentrated on the rendering side and cut hard there. A path where many players share the same world state while each sees it from their own viewpoint also opens up naturally from this structure.
The renderer has to learn the mapping "plain shape frame → high-fidelity video." For that it needs paired data: the same scene captured in two versions. One is the finished, high-fidelity video; the other is the shape frame at the same camera and the same timing. The two have to line up pixel for pixel so the model can learn the rule "this skeleton gets this texture."
This paired data was synthesized in Unreal Engine 5. One thing to flag: the Unreal Engine 5 that made the data and the "lightweight game engine" that holds state and rules during play are similar only in name — they serve different purposes. One is the studio that shoots the training material; the other is the stage where the game actually runs.
Four kinds of scenes were covered — megacity driving, third-person shooting across a desert expanse and a medieval castle, and aerial combat in a desert canyon — for a total of 114,334 clips and 1,712 hours. (Per-scene scale and the data held out for evaluation are collected in the appendix.)
The shape frames look like this. The sky is black, the floor a gray checkerboard, buildings and walls are solid-color boxes and prisms, trees are a sphere on top of a cylinder, characters are blocky figures. Everything is outlined in black only, with no texture, reflection, or shadow. It amounts to keeping only the geometric structure of the space and stripping away the rest. Along with these frames, the engine also records object IDs, visibility, and interaction events.
That record feeds into the "dynamic captions." This is exactly the "description" we deferred earlier in the real-time loop. Dynamic captions are made without calling any model — from the engine's state values alone. In an interactive game, the screen changes every time the player moves. Turn the camera and the visible objects change; events change object states. A single fixed line of description cannot keep up. So the meaning description pre-attached to each object (that is the description the agent prepared) gets filled out with the currently visible objects and their real-time state through a rule-based template. When a car drives off, it drops out of the description; a pedestrian flips from "standing" to "approaching"; a newly entered object is marked "changed." The shape frame supplies the structure — position and layout — and the caption supplies the meaning — which object is in which state — each filling in the other. Because it never passes through a vision-language model, there is little room for error and no added latency.
The rendering model is a Fun Control variant of Wan2.2 5B reworked into a video-to-video (V2V) model. V2V refers to taking an input video and turning it into a differently-looking video. When generating one chunk, four things come in as conditions. The engine's shape frame dictates "what to draw where," the dynamic caption supplies the details, the spatial-memory reference frame keeps things from drifting away from what was seen earlier when the player returns to the same place, and the first frame stitches the newly made video onto the current screen. (Network specs and training settings are in the appendix.)
Real-time is a tug-of-war between speed and responsiveness. Generate in short bursts and response gets faster, but you have to call the model more often, and if each call does little work the GPU idles waiting for the next instruction. The paper resolves this tug-of-war with two moves — to put the effect plainly, one is "draw fewer times" and the other is "reuse computation already done."
Draw fewer times. A diffusion model normally completes a picture by peeling away noise little by little over many steps. This model compresses that process into one or two steps via distillation. Distillation means teaching a smaller procedure to mimic the output of a larger model. Fewer peeling steps means proportionally less computation per frame, hence faster.
Reuse computation already done. Instead of computing everything from scratch each frame, it stores and re-pulls the intermediate results of earlier frames and of captions that have not changed. Captions in particular are recomputed only when they change. The time saved is exactly the time not spent computing the same thing twice.
The biggest share of the cumulative optimization came from two steps. Distillation that cut generation to a single step pushed throughput from 24.2 to 51.2 FPS, and kernel optimization that fuses multiple operations into one dropped latency from 367ms to 121ms. The step-by-step stacking of the optimizations and the numbers at each stage are collected in the appendix table. Overall it runs about 19x faster and latency drops about 9.9x.
The numbers have to be read with their conditions, or they mislead. In particular, do not mix the two kinds of speed. One is the throughput at which the model stamps out frames; the other is how fast the screen responds when the player presses a key. They measure different things.
What the paper puts forward as its headline result is the response speed. On a single RTX 5090 it plays at 480p and 60FPS, and it takes 112ms for a keypress to show up on screen. That is the number tied directly to how play feels.
The headline throughput figure is 83FPS, the steady-state generation speed of single-step denoising. GPU memory here peaks at about 18.2GiB. This is why 60FPS play and 83FPS throughput are listed separately. The former is the frame rate at which the game actually runs; the latter is the speed at which the model churns out frames.
It stays usable as you step down to consumer GPUs. With a fixed prompt, single-step generation reaches about 67FPS on an RTX 5080 and about 58FPS on an RTX 5070 Ti. Both are 16GiB-class, which widens the range for local deployment. But do not let the qualifier "fixed prompt" slip by. As we saw, when the caption changes the text-side computation has to be redone, so a fixed prompt has the speed advantage. The design that caches captions shows up directly as a performance condition.
On the feature side, it demonstrates editable shooter and driving games where the engine enforces the rules. Releasing the code, the model weights, and the low-poly assets is useful for both research and creative work.
Start with the constraints the authors state. Training uses one model per game. Each of the four scenes has its own dedicated model; a single general-purpose renderer does not handle all the games. The data is also entirely paired video synthesized in Unreal Engine 5, so the shape-to-high-fidelity mapping is defined within the range this synthesis pipeline can produce. The high FPS on consumer GPUs also presumes a fixed prompt.
A deeper question appears once you unpack the phrase "the world is consistent." It is actually three different demands.
First, state persists. A drum that has blown up once has to stay gone. This is largely solved just by the engine holding state in code.
Second, the same place looks the same. Even if the engine holds the fact that "there is a warehouse here," redrawing that warehouse is the model's job. If the model draws it a little differently each time, the state is unchanged but the appearance wavers. The reason for keeping a separate spatial-memory reference frame is precisely to pin down this wavering of appearance. The official model card is candid that this wavering is not fully solved — appearance drift and geometric errors still surface over long runs, under occlusion, and during fast camera moves.
Third, adjudication and picture agree. If the engine ruled a hit but the screen makes the bullet look like a miss, the player has no idea what to believe. If a door the agent made up is registered in the engine as "non-openable decoration" while the picture renders it as if it opens, the player who walks over grasps at nothing.
It is too early to declare this a confirmed defect. But we can say how to look into it. Leave a place and return, and measure how much the appearance has changed; compare, frame by frame, whether the engine's hit ruling and the hit effect drawn on screen line up. How consistently the flow of inventing scenes and rules in natural language actually lands as code is, for now, confirmed only through qualitative demos. The moment you measure these three separately, a design introduction deepens into a point of view for evaluation.
The principle we take from this paper is a single one: manage persistent state outside the model's conversational memory. This is not a description of something already built in ONTLE — it is a view of this paper held up against our own work.
The problem we are wrestling with right now is this. An AI PM tracks the work status of several projects over a week and builds a briefing every morning. The common failure is to make the model sustain state by telling it to "remember last week's context." When a PM agent reasons "was this task finished?" from conversational memory alone, a small slip bleeds into the next briefing, and the one after that. It is the same trap as the structure where a video model holds world state in pixels and context and lets error accumulate.
So the design we apply is the same. The truth of state lives not in the model but in the task DB. Which task is in which state is held by a deterministic system, and even when that task is outside the current conversation context, its state remains in the DB. The model moves the state the DB hands over into a human-readable briefing, and translates a request like "just the ones delayed this week, framed around risk" into query conditions and classification rules. It is the same spirit as deriving the dynamic captions with a rule base.
But there is a point where the analogy does not carry over cleanly. A game engine creates the world's state itself, so the values the engine holds are true by construction. A work DB is different. Task state is filled in only when external signals come in — commit logs, review approvals, a person's report — and those signals are trustworthy only after interpretation and confirmation. The DB "storing" a state and that state "coming in and being updated correctly" are different problems. So we add one more step. Tasks whose completion is ambiguous or whose sources conflict — say, a commit is in but the review is unapproved — are not auto-reflected into the briefing; they are split off as "needs confirmation" and only cross over to "done" after a person or the original source is asked once and the state is updated.
It is a research-stage technical report, and it releases the code, the model weights, and the low-poly game assets. The models and games the official repository has released are three — medieval castle, ancient Egypt, and city driving (the desert-canyon aerial combat used in training is not on the released list) — and because the structure trains a dedicated model per scene, it does not extend straight out of the released scenes.
Not yet. It presumes that the data is paired video synthesized in UE5 and that the structure trains a separate model per scene. The authors position this work as a "concrete preview" of how coding agents and neural rendering might be used in a game development workflow.
Table 1. Training data
| Scene | Clips | Duration | FPS |
|---|---|---|---|
| Medieval castle | 17,848 | 290 hours | 24 |
| Ancient Egyptian desert | 40,296 | 643 hours | 24 |
| Desert canyon aerial combat | 28,720 | 467 hours | 24 |
| CitySample megacity | 27,470 | 312 hours | 17 |
| Total | 114,334 | 1,712 hours |
The shooter and aerial-combat clips are roughly a minute long. To guard against overfitting and evaluation contamination, three complete capture sessions each are held out for the castle, desert, and city, and the entire multi-machine capture batch is held out for the aerial combat, used for evaluation only. Not a single clip from these sessions goes into training.
Table 2. Cumulative effect of optimizations (RTX 5090, chunk = 1 latent frame = 4 frames, KV window = 21 latent frames)
| Optimization (cumulative) | FPS | Latency |
|---|---|---|
| Baseline Causal Wan2.2 DiT (4 steps, full Wan VAE) | 4.5 | 1110ms |
| + Tiny VAE (TAEHV) | 6.5 | |
| + RoPE kernel, text KV cache | 15.4 | |
| + 8-bit weights, CUDA Graph | 24.2 | |
| + 1-step distillation | 51.2 | |
| + Async I/O pipeline | 57.7 | |
| + Fused Triton kernels | 65.8 | 367ms → 121ms |
| + Ring KV cache | 73.7 | |
| + Split-KV FlashAttention | 83.4 |
Network (DiT, 5B parameters)
Training (one model per game)
[^1]: The authors liken this "transposition into picture" to intersemiotic translation in Jakobson's classification of signs. The side that holds the meaning is the upstream language model, and the diffusion model transposes its instructions into picture.