When we pick what to hand a generative model as a guide, we usually reach for the input that looks sharper and more precise. A piece of research called LEGO flips that instinct on its head. A picture that is blurry but sits in the right place turns out to be more useful than one that is crisp but misaligned. Why that is, and where the principle holds beyond video, is what this whole piece is about.
Start with the task LEGO tackles. Someone is cooking in a room. A camera mounted on the wall records that person from the outside (third person), giving us one video. What we want to build is the video that person would have seen (first person). All we're given is the path the target first-person camera moved along.
The problem is that the two viewpoints barely overlap. The third-person camera sees the person's back and hands. The first-person view takes in the cutting board, the ingredients, and the far wall. A large portion of the footage we need to produce was never captured in the source at all. There's nothing to copy, so we have to invent what isn't there. That makes this one of the harder flavors of "render the scene from a new viewpoint."
The best-performing approach so far (EgoX, published in 2026) took the head-on route. It estimates the distance to each point (depth) from the single camera's video alone, and uses that distance to resurrect the third-person video as points in 3D space. It then redraws those points from along the first-person path and feeds that to the generative model as a guide. Because each pixel is moved to exactly one location according to its estimated distance, the texture stays crisp.
That's also where the trap is. When the distance estimate is wrong, the content of that pixel gets moved to the wrong place crisply. Sharply, and misplaced without leaving any signal about where it went wrong. In statistics, this kind of error — stamping something down with confidence in the wrong spot — is called bias. It's the expensive kind of mistake, one that leaves no thread to pull on to undo it.
Here's a scene to build intuition. There's a painter who is phenomenal at filling in detail but poor at fixing composition. You hand this painter one of two rough sketches. One is a photo where everything is crisp but the objects sit slightly out of place; the other is a charcoal sketch where the outlines are smudged but everything is where it belongs. Which is the better one to hand over?
The answer is the sketch. The painter is good at filling smudged outlines with detail, but can't fix a misplaced position on their own. What they need is "where is what." "How it should look" they'll fill in themselves.
LEGO's starting point is that a generative model is just like this painter. If so, the criteria for choosing a guide change. Alignment (what is where) comes first; sharpness (how crisp) comes second. And this is where we can see again why the point-cloud approach above makes such an expensive mistake. That approach, in effect, gave up alignment to preserve sharpness.
With the principle set, let's look at the implementation. LEGO is "lifting-free" — it strips out the whole reconstruction step that lifts pixels into a point cloud. In its place sits a single view synthesizer: a neural network trained to take an input frame plus "how it should look from the target viewpoint" and draw a new frame.
The key here is how the 3D information gets passed along. Instead of distance or a point cloud, LEGO hands over a kind of viewing-angle map that records, for each pixel of the first-person frame, "which direction it should be looking." This map encodes, per pixel, where the camera is and which way it faces. Because every direction is written relative to the third-person camera, the actual distance between the two cameras falls naturally into the map as well. The model is fine-tuned on many paired third-person and first-person videos, and the only supervision signal is the target first-person frame itself — no distance labels, no correspondence labels between the two views. And yet, guided only by the goal of "render this frame," the model learns the correspondence between the two viewpoints on its own, internally.
Why is the picture this synthesizer draws blurry? Where it isn't confident, the candidates for "where to pull from" straddle several locations, and the model averages over those candidates. The texture gets smudged, but the content stays near its correct place. If the point-cloud approach's mistake is "content landing somewhere wrong," the synthesizer's mistake is "blur." And blur is exactly the kind of damage a generative model has already been trained to undo as it strips away noise.
The synthesizer throws in one more signal for free. How tightly the candidate locations cluster around a single point is, in effect, a per-location confidence. Tightly clustered means trustworthy; spread wide means the model isn't sure where to pull from either. LEGO uses this confidence in two places. First, low-confidence regions are masked out to a neutral gray in the guide. The point-cloud approach leaves holes where it has no basis, but an averaging approach fills in even where there's no evidence, so it can't say "I don't know" on its own. Confidence supplies that missing voice. Second, the same confidence is pulled back out during generation to tug the model toward the trustworthy side in the early stage where the frame's layout is being decided. This device, which needs no additional training at all, is ARC (Adaptive Rendering Consistency).
Adapting to the first-person camera is quick work. The publicly available synthesizer was trained to interpolate between ordinary cameras close to one another, but LEGO's target is the fisheye headset camera that bends widely at a wide angle. Since all the camera information comes in through the viewing-angle map, there's no need to change the model architecture — you just invert the fisheye distortion per pixel to recover the target direction, and re-tune the weights. This fine-tuning finishes in about 3.5 hours on four NVIDIA H200 GPUs for 10,000 steps. After that the weights are frozen.
To trust the claim, you have to look at the experimental design. Both training and evaluation follow EgoX's procedure exactly on the Ego-Exo4D dataset. The test set is split into scenes that appeared in training (seen) and scenes that didn't (unseen). This split matters because it separates whether the model memorized the scenes it saw or actually learned viewpoint conversion itself.
The metrics mix things of different character, so here's a one-line primer before the table.
The comparison was made fair with two mechanisms. In the experiment that measures each component's contribution by removing them one at a time, the backbone, training setup, random seed, and camera path were all kept identical, and only the guide input and the way generation is steered were changed. They also report EgoX's scores in two versions: one is the number reported in the original paper (EgoX), the other is the publicly released EgoX model re-run through the authors' own scoring code (EgoX†). In other words, they didn't hide the fact that the same model yields different numbers under different scoring methods.
Start with the overall comparison.
| Split | Method | PSNR↑ | SSIM↑ | LPIPS↓ | CLIP-I↑ | FVD↓ |
|---|---|---|---|---|---|---|
| seen | LEGO | 20.28 | 0.689 | 0.310 | 0.925 | 155.56 |
| seen | EgoX | 16.05 | 0.556 | 0.498 | 0.896 | 184.47 |
| seen | EgoX† | 15.57 | 0.556 | 0.466 | 0.903 | 197.95 |
| unseen | LEGO | 15.91 | 0.505 | 0.504 | 0.897 | 404.77 |
| unseen | EgoX | 14.38 | 0.457 | 0.552 | 0.877 | 440.64 |
| unseen | EgoX† | 13.53 | 0.437 | 0.542 | 0.886 | 455.06 |
LEGO leads on every image metric and on FVD. The authors note that the gap is largest on PSNR and LPIPS. The one worth watching here is LPIPS. Leading on that metric can be read as getting closer to the ground truth not just in pixel-level numbers but in the perceptual quality a human eye feels. That said, the metrics that measure flicker or motion smoothness were near saturation for every method and had no discriminating power, so this contest was won by "drawing the right frame," not by "rendering it smoothly."
In the previous section, "alignment first" was stated as a principle. In this section let's see how that principle is confirmed by experiment. The experiment that adds components one at a time comes closest to that evidence. Generating with no guide at all lands PSNR at just 12.30. Feeding a point-cloud render as the guide raises it to 15.59, and feeding the fine-tuned synthesizer's render raises it to 18.24. The decisive contrast is the render from the synthesizer without fine-tuning. That one is 15.71 — essentially the same as the point-cloud render (15.59). This reads as evidence that the synthesizer's gain comes not from the transformer architecture but from the fine-tuning that adapts it to the first-person fisheye camera.
The experiment that takes aim at the hypothesis most directly is the one that measures the quality of the guide itself. Here they compare the "guide picture" directly against the ground truth, not the generated result. Sharpness is measured as the ratio of crispness relative to the ground truth: the synthesizer render scores 0.28, the blurrier of the two (the original being 1). The point-cloud render scores 0.97, nearly on par with the original. But on alignment — how well the content at a given location matches — the order flips. The blurrier synthesizer render has the better alignment of the two, while the point-cloud render, crisp to near-original levels, has terrible alignment. The results followed alignment too: the blurrier one guided the model better. And the guide's blur didn't even carry into the output — the generations from the two renders ended up at the same sharpness. The synthesizer even wins on speed: 4.7 seconds per clip, far cheaper than the 69.2 seconds of depth estimation plus point-cloud rendering.
That even a good technique only works on top of a good guide shows up in the comparison of steering methods. Turning on ARC improves every metric with no retraining. By contrast, layering on the depth-based guidance EgoX used makes FVD worse, at 194.98. Attaching ARC to the point-cloud render does worse than no guidance at all (PSNR 13.96, FVD 735.50). ARC is a device that reinforces the render it latches onto, so it only earns its keep when both an aligned render and a trustworthy confidence are present.
Generalization was confirmed on the EgoHumans and Nymeria datasets. Applying the Ego-Exo4D-trained weights without touching them, LEGO still beat EgoX† on both, across every metric. To the suspicion of overfitting, the authors point out that even with entirely different scenes and people, the fraction of render the confidence gate keeps stays around 15% — similar to unseen Ego-Exo4D. That gives grounds to read the fine-tuning as having adapted not to a particular training environment but to a shared property: "fisheye first-person cameras with large viewpoint changes."
The authors address limitations separately in the appendix, but even from the released excerpts the structural tensions can be pointed out.
Adapting it costs something. Fitting the public synthesizer to the fisheye first-person camera takes a one-time run of about 3.5 hours on four H200s. That said, this is a one-time setup cost. The paper applied these adapted weights to the new environments of EgoHumans and Nymeria with no retraining. Whether a wholly different optical system would require re-adapting, and how much that would cost, was not measured.
The figure that the confidence gate keeps about 15% of the render does not mean the rest is generated purely with no guide at all. What gets masked out at low-confidence locations is only one kind of input — the synthesizer render — while the generator still receives, at those same locations, the original third-person tokens and the text condition. ARC, too, is a device that continuously reflects the pre-masking render according to confidence, so it doesn't cleanly switch trustworthy and untrustworthy locations on and off. So there's no basis for reading the retention rate directly as "the fraction of the area where hallucination can occur." Still, given that the task has almost no overlap by its very nature, the share the model must fill in itself is large, and the tension between "marking it as unknown" and "filling it in correctly" remains.
The scope of transfer is also open. The fine-tuning is fitted to the fisheye first-person camera. The authors argue that since the adaptation targets the shared factors of camera model and viewpoint change, it transfers beyond the dataset — but transfer to a wholly different optical system, such as an ordinary narrow-angle first-person camera, was not confirmed.
Finally, this entire logic rests on the premise that "the model revives blur into detail." That premise holds only when the guide is correctly aligned. At extreme viewpoints where alignment itself breaks down, there remains a risk that ARC reinforces a wrong render instead. The result that scores got worse when ARC was attached to the point-cloud render shows, in reverse, just how sensitive this is.
So far it's been a story of cameras and pixels. But is "alignment first" tied only to problems of space? For any task where you take incomplete context as a guide and have to invent the rest, the same questions apply: whether to feed in a crisp single answer or a rough sketch that only has the positions right, and how to leave blank the places you're not sure about. The AI assistant ONTLE builds does exactly this kind of work.
Let's follow one concrete scene all the way through. A directive comes in over Slack to ONTLE's AI PM (an AI manager that breaks work down and schedules it on a person's behalf): "Four cards for this card-news set, same tone as last time, sometime next week." It's a sentence that works between people, but it's full of blanks for a machine to turn into an executable task list and schedule. At this juncture, RAG (a technique that searches past records and attaches them to the current context) typically finds similar past work and hands it to the language-generating AI (LLM) as a guide.
The common approach is to fill that guide in "precisely." It picks the single most similar past directive and copies its schedule, owner, and deliverables straight in. This resembles the trap the point-cloud render fell into. If that past case is subtly off from the current situation, the wrong owner and the wrong deadline get stamped into the draft with confidence — with no signal at all about where it went wrong.
Flipping it the LEGO way goes like this. Instead of feeding the search result in as a crisp single answer, you feed it in as a "structural sketch" averaged across several past cases. Even if the specific dates and names get smudged, the skeleton — "card news goes planning, image generation, editing, distribution, and usually splits into two or three stages" — stays in place. And where search confidence is low — say, a new task with thin past cases to reference — you leave it grayed out like confidence gating. Instead of being confident about an invented date, you leave it "to be determined."
Here the limits of the analogy have to be made clear. Spatial alignment in video and contextual fit in work are not the same problem. So this principle must not be translated into "invent the answer better." Quite the opposite. When there's not enough basis for the owner and the deadline, you don't lock them in. You stand up the frame first and agree on it with a person, and only the cells marked empty get filled in after human confirmation. This is, in fact, how the ONTLE workflow already operates.
We'll be clear that this is not a proven design but a hypothesis we intend to test. We mean to actually measure whether reworking the AI PM's draft generation around the principle of alignment first, sharpness second really reduces hallucination.
No matter how much you improve depth, it's the *nature* of the remaining error that's the problem. In the point-cloud approach, the leftover distance error turns directly into a "confident positional error." This kind of mistake, which leaves no signal to undo it, doesn't vanish just because you push estimation accuracy up. LEGO, in effect, changed the nature of the error rather than competing on accuracy — toward one that may blur from averaging but keeps the position near its correct place.
It does. But that's a different (easier) task. Early systems leaned on privileged inputs like a ring of surrounding cameras or the first frame of the ground-truth first-person video. The setting LEGO targets is a single third-person video only. Not requiring extra input is a premise of this method.
Scores dropping on scenes not in training is to be expected, but even over that unseen range LEGO still beat its comparison targets. The more important evidence is that it held its scores on the EgoHumans and Nymeria datasets — not used in training — without touching the weights. That said, this is all within the shared condition of a fisheye first-person camera.