Picture smart glasses recording everything in front of you, all day long. After a few years, that's a pile of hundreds of hours of video. One day you ask: "Where did I put my passport?" The answer is in there somewhere, for sure. But you can't rewind through hundreds of hours. The cost of finding it would dwarf the value of the answer.
MemLife, from researchers at Meta Reality Labs, the University of Virginia, and the University of Illinois, solves this differently. Instead of combing through the entire video, it jots down what it sees as it sees it — short written "memories" — and when a question comes in, it pulls only the fragments it needs from that pile of notes to answer. Layered on top, MemOpt goes one step further: it lets the machine practice, on its own, what to write and how to write it so it can be found later.
Here's the headline. When the system was taught only how to write memories, accuracy went up (from 56.5 to 60.4 on one benchmark). But once the side that reads the memories and produces answers was already smart enough, that extra gain noticeably shrank. That's the most interesting thing about this work.
The reason: this system has two players. One writes good memories (call it the writer), and one reads those memories to extract an answer (call it the reader). The two are locked in a tug-of-war where each can partly substitute for the other. A sloppy write-up can be salvaged by a smart reader, and a mediocre reader can still coast on well-written memories. This paper digs into the "how to train the writer" side of that pair. Let's follow how far it actually gets.
Finding an answer in egocentric video (footage shot from your own point of view) runs into three problems at once.
First, it's too big. The amount an LLM can read in one go — its context window — is fixed. You can't cram hundreds of hours of video frames into it. And re-processing the raw footage for every question is unaffordable in both time and compute.
Second, it's too vague. So you compress the video into text and store that. But cut too much and the very detail a question asks about disappears; write down everything and trivial minutiae muddy the search. In the worst case, the writing side invents things that aren't in the video (hallucination) and plants fake evidence.
Third, it's too similar. Yesterday's kitchen and today's kitchen look almost identical. The more near-identical memories pile up, the harder it gets to tell which occurrence is the right one, and the correct answer gets buried. The paper calls this retrieval competition. It's the most insidious wall — one that only reveals itself once the store of memories grows large.
So the fight splits in two: either you leave out key evidence while writing (a write failure), or you fail to pull the right item while searching (a retrieval failure). You have to win both to get the answer right. Yet in practice, if you grade everything with a single "accuracy" number, you can never tell whether the thing to fix is the writing side or the searching side. That's exactly why the protagonists of this story treat the two separately.
MemLife first builds a "good writer" using human-authored rules. There are three principles.
There's one more clever choice. The writer produces a description for each video clip independently. Because it doesn't lean on the result of the previous segment, one mistake doesn't bleed into later ones, and the workload and storage grow in exact proportion to video length (twice as long, exactly twice as much). They also tried letting it reference the preceding context, but accuracy didn't improve.
The retrieval side is the reader. The reader doesn't settle for one search per question — it's an agent that works through steps on its own to find the answer (an agent is a program that handles a task by choosing its own next action). When "Where did I put my passport?" comes in, it moves roughly like this. First it reshapes the question for search, locking onto the cue "passport" and a plausible time window. It scans similar memories in that window, pulls the most promising passage in full, and reads it. If the original video is still around, it checks a few frames of that scene directly, and then it answers. The memories it pulls are handed over in chronological order — because when you retrace a long story, the order itself is a clue.
Up to here, everything is human-authored rules. MemOpt parts ways here.
Ordinary reinforcement learning (training a habit by rewarding good actions and penalizing bad ones) hands out reward based on "was the final answer right." But if you tie reward to the final answer, how the reader wielded its tools gets mixed into the signal. You want to tune the writing, but the noise of the reader's execution process comes along with it. Imitation- and distillation-style approaches (learning by copying a stronger model's behavior) adapt poorly, and final-answer-reward approaches can't escape this noise.
MemOpt freezes the reader and places reward and penalty solely on "how the memory is written." You can picture the mechanism like this. The finished memory and the original video are handed together to a frozen model (the same model, not trained further) with one instruction: reproduce whatever the video actually supports as-is, but revise the parts it doesn't. The score then comes from comparing, token by token, how much the model's probabilities shift under that instruction. Where the memory matches the grounded content, little needs changing and the score stays high; where the writer invented something not in the video, the model rewrites it and the probability gap drags the score down. Because this is a continuous, per-token measure, fabrication lowers the score rather than zeroing it out — there's no hard guarantee that a made-up memory earns nothing at all.
The reward is named FIRM, and it applies three yardsticks.
The three scores aren't added — they're multiplied. If any one approaches zero, the whole thing collapses. A memory that's faithful but never surfaces earns almost nothing, and one riddled with fabrication sees its faithfulness score — and with it the whole product — pushed down. Only a memory that satisfies all three at once is a good memory.
Checking retrievability every time means running the reader each time, which is expensive. So at the start of each training cycle, the reader is run once over all the training questions to record "how it searches," and after that, that recording is replayed just to check whether a memory would surface. It uses a publicly available reinforcement learning algorithm, with added handling to keep lengths even so that long memories don't hog the score computation.
The core point is one thing: without touching the reader, reward and penalty are drawn only from "verified evidence, the original video, and frozen search behavior." It strips the reader's noise out of writer training entirely.
MemLife alone, with no training, already beats the existing "training-free" methods on all four benchmarks. Add a trained writer — MemLife+MemOpt — and it beats all the "training-based" methods too.
| Method | SuperMemory-VQA | EgoLifeQA | SuperMemory-LVQA | EgoLife-EQA |
|---|---|---|---|---|
| EgoRAG (strongest training-free baseline) | 49.28 | 48.20 | — | — |
| MemLife (no training) | 56.50 | 52.80 | 56.18 | 52.00 |
| MemLife-V (also references the original video) | 57.78 | 53.20 | 57.95 | 50.00 |
| MemLife + MemOpt (trained writer) | 60.35 | 56.60 | 58.91 | 57.00 |
| Oracle Context (reference ceiling: the ground-truth original-video evidence fed in directly) | 67.58 | 66.20 | — | 61.00 |
The numbers in the table are accuracy, and higher is better. All four benchmarks are question-and-answer tests about old egocentric video. The last two are variants: SuperMemory-LVQA merges all ten people's records and mixes in "other people's records" as distractors to deliberately enlarge the search space — a stress test — while EgoLife-EQA asks about recurring events, with the grounds verified by hand against the original. The paper tallies the improvement over the strongest prior method at 4.0–17.0%, and reports shrinking memory size by up to 31×.
The real achievement is transfer (transfer means it works even on data it wasn't trained on). MemOpt was trained on only a subset of SuperMemory-VQA subjects, and the other three benchmarks were never used in training. Yet the scores rise there too. That means it holds up even when the kind of video, the type of question, and the scale of memory all change.
Two rows in the table are worth a close look. One is Oracle Context. This is a reference setting that skips the retrieval step and feeds the reader the ground-truth original-video evidence directly (not the text memory the writer wrote, but the original-video evidence that actually grounds the answer). Think of it as the upper-bound reference for what happens when retrieval is ideal — the variable of search pulling up the wrong occurrence is removed. The fact that it still falls short of a perfect score even when handed the grounding evidence means there's still room left in the reading itself — in extracting the answer from that evidence.
The other is MemLife-V. This is the setting that goes back and looks at the original video again, and it's barely different from MemLife, which uses text only. That matters. It means the credit for the improvement lies not in expensive re-checking of the original video, but in the well-written "text memory" itself. If you don't have to reopen the video every time, you can dramatically lower the operating cost of a memory system.
How training reshapes memories is vivid in the examples. Before training, the writer mislabeled lentils as corn; MemOpt leaves the surrounding action (moving toward the sink) untouched and fixes only the object. A memory that blurred things with vague pronouns and omitted the food gets corrected to pin down the person and insert "roasted chicken" explicitly. Pare away the invented, sharpen the necessary — faithfulness and informativeness rising together.
Here the tug-of-war from the opening shows up in numbers. Use a smarter reader and the absolute scores jump a lot. On SuperMemory-VQA, upgrading the reader takes 56.50 all the way to 66.93. But the extra you earn by adding MemOpt on top of that is just over one point — from 66.93 to 67.90.
That's because a smart reader fills in the gaps of flimsy memories on its own. The effort of writing good memories and the ability to read cleverly are in a tug-of-war: grow one side and the other side's share shrinks. So "should I invest in writer training, or use a better reader model" is not a question with one permanent right answer. The resources and costs you have decide it.
The line the authors draw for themselves is honest.
Recall (the proportion of ground-truth evidence actually dredged up) is uneven. Even when MemOpt raises accuracy, recall sometimes goes up and sometimes down. The authors see the accuracy gain as coming not only from "dredging up more evidence" but also from "packing more useful memory for answering." It's a self-caution that you can't conclude search improved just because accuracy did.
The cost structure isn't light, either. Scoring retrievability rests on a procedure that runs the reader over all training questions each cycle to gather its behavior. They save with replay, but that collection itself can be sensitive to scale. On top of that, both faithfulness and informativeness scoring are handled by the writer itself (the same model, frozen). Since it's grading its own output, the kinds of hallucination that model can't see have room to leak through in the reward signal too. And it's worth remembering that even the verified transfer is still within a single genre — egocentric video QA.
This isn't someone else's story. It cuts to the heart of a problem ONTLE's AI PMs run into every day.
On a single customer project, a PM persona stays attached for months — taking requests, holding meetings, making decisions, delivering outputs. Those months of conversations, logs, and deliverables are exactly that "pile of hundreds of hours of video." Let's follow one concrete scene all the way through.
One day in September, the client contact, Mr. Kim, says the PG fees are too burdensome and asks to drop the payment integration from the first-phase scope. After the meeting, the PM agrees to it. Then, two months later, the client asks: "That decision to not add the payment integration — when and why did we decide that?"
Under the current setup, the PM scrapes together two months of logs for every question and shoves the whole thing in. The longer the project, the more directly it hits the very wall MemLife flagged. It's too big, so re-processing is expensive, and because there were several similar meetings, the wrong occurrence gets dragged up.
Apply the MemLife-style prescription and the picture changes. Every time the PM finishes an interaction, it writes that session down as a single-line memory stamped with time and entities. "On September 10, in the payment-integration meeting, on the grounds of contact Mr. Kim's concern over PG fees, I dropped payment from the first-phase scope." The key is that this is not a summary written with foreknowledge of the future question. By embedding entities — contact, date, grounds — a question that arrives two months later can be narrowed down to that date window and pull exactly that memory.
MemOpt's implication runs one layer deeper. What MemOpt did was define "what makes a good memory" with three yardsticks and nudge the writing toward them. Carried over to ONTLE, it means the company holds the definition of this "good memory" in its own hands. Deciding which decision's grounds must be kept, and what to write down as important, is the definition of recording quality. But at ONTLE, faithfulness weighs especially heavy. If a PM memory fabricates a fact, the trust in the customer's deliverables is immediately eroded. So the blind spot of a structure where a model grades its own records is, for ONTLE, exactly the place to fill with cross-verification by a different model.
Rather than memorizing someone else's benchmark numbers, it's more useful to diagnose your own memory system right now.
No. When MemOpt was used to optimize EgoRAG's writer, EgoRAG's own base pipeline improved on two benchmarks, and reading those memories with the MemLife reader pushed it higher still. Even swapping the underlying base language model from Qwen3.5-9B to Qwen3.6-27B kept the improvement across every combination. In other words, it isn't a technique tied to one specific model.
Adding them one at a time, on SuperMemory-VQA: reading picture and sound together gave 52.33, attaching names to things 54.09, and adding first-person narration 56.50 (before training). On the reader side too, narrowing the search range by time and sorting chronologically raised EgoLifeQA from 50.40 to 50.60 to 52.80. That said, recall isn't monotonically increasing — not every device pushes every metric in the same direction.
How many answer choices these benchmarks have isn't in the public excerpt, so I won't assert a random-guess baseline. But even the reference ceiling — feeding in the ground-truth original-video evidence directly — stays in the 60s. That means there's a ceiling even when you assume ideal retrieval and hand over the grounding evidence directly, and the remaining share belongs to the reading side that extracts the answer from that evidence.