Picture a night control room. The operator watches the monitors, and every time something important passes by, they jot a line in a notebook beside them, noting the time. Much later, when someone asks, "What was that thing earlier?", instead of rewinding the tape they look at the current screen alongside that notebook. The notes alone bring the old context back to life.
This is exactly what 'OneStreamer' — a streaming (continuously, in real time) video AI released by Nanjing University and collaborators — does. It writes out the scenes it observes as timestamped captions of its own, short descriptions noting 'what it saw and when,' and later pulls those notes out to answer questions. Built on Qwen3-VL-4B, it's a small 4-billion-parameter model, yet it beat every competitor across eight tests of streaming video understanding. The secret behind how something this small did this well lies in that one 'notebook.'
The pain of real-time video AI is nastier than it sounds. Think of a security guard watching CCTV, or a wearable assistant device. Whether a scene that just went by will matter later is impossible to know in that moment. A scene that looked trivial can turn out, ten minutes on, to be the key clue to a question. But by then, that scene has already been pushed out of the model's narrow working memory — its context — and is gone.
The responses so far have fallen into roughly three camps, each with its own weakness.
First, re-retrieving the past. Compress past video or search and pull it back in. The trouble is that the old video fragments you hold onto compete with the scenes coming in now for the same scarce context slots. The more of the past you grip, the blurrier your view of the present becomes.
Second, continuously updating the reasoning. As the video flows, the model keeps rewriting its running thought of "this is what's happening now." But the trace of 'here's how I judged it' and the factual record of 'what was there and when' are different things. A trace of reasoning doesn't make for a ledger of facts you can later pull out as-is.
Third, using a special signal to control "answer or not." Here lurks a data-imbalance trap. For most moments of a video the model is "still quietly watching," and the moments when it actually needs to output something are rare. Train on all these moments equally and the model over-learns "silence."
The core question narrows to one: how do you turn incoming observations into 'fact memory you can pull out again,' so they feed into well-timed responses that draw on evidence from both the past and the present?
OneStreamer's design boils down to one sentence: it learns when to speak and what to say together, within a single unbroken stream.
Normally, when we say an AI 'generates' text, we mean it produces an answer to show the user. OneStreamer applies that same generative ability to writing 'observation notes' the user never sees. Instead of piling up video fragments to preserve the past, it transcribes it into text in the moment of observing. Not retrieval, but recording. Text is far lighter than video fragments, so it takes up less room.
The device that plays this notebook role is PHCM (a hierarchical memory that writes observations down). It writes at two granularities. There's a detailed record that densely transcribes short segments, and a sparse summary that bundles completed events into coarser time units. Each record is tied to the timestamp of the source segment that backs it, forming a 'detail ↔ event-summary' hierarchy.
The clever part is that the video input itself is fixed to a 'recent window' — a window holding only the last few scenes, with earlier ones pushed out in turn as new ones arrive. It doesn't grow this window to grip the past. The past is carried in text notes alone. So the model holds the context of the distant past while its view of the current scene stays unnarrowed. The title's 'without clouding the present' refers to exactly this — the field of view on the current scene. It does not refer to response speed itself. That said, the paper does measure response latency — in time-to-first-token (TTFT, §5.3, Table 4), PHCM comes in at 0.124s and FIFO at 0.094s, far faster than the 4.560s of Full, which carries the entire past video. Note, though, that this figure measures only the response stage of pulling out notes written in advance, so it should be read separately from the end-to-end processing latency from observation to response.
The second axis is how it teaches 'when to speak.' The principle is one sentence: common silence is taught only in part, while rare but important actions — the moments of leaving a note or starting a response — are taught without exception, every single one. This suppresses silence's numerical dominance and keeps the model from forever just reading the room.
The control signals work like this: quietly observing with no record and no response; leaving a memory note; a wait state meaning "evidence is surfacing but I'm not ready to answer yet"; and the answer the user sees. These four alternate within the same single stream.
The most interesting result in this paper isn't the aggregate score — it's the point where it broke the old past-versus-present trade-off. Until now, the more of the past you carried, the worse your judgment of the current scene got. By carrying the past as text rather than as video, OneStreamer largely sidestepped that trade-off.
How did they confirm this? They fixed the same checkpoint and the same 'recent 16 scenes' window, and changed just one thing in the comparison — whether or not the captions written during observation are carried forward into later predictions. Three variants are pitted against each other: discarding the oldest first and keeping no notes (FIFO), retaining the entire past video (Full), and OneStreamer's way of carrying the past as caption notes (PHCM).
Start with the present side. PHCM, carrying notes, didn't shave its current-perception score even below note-less FIFO. OVOBench real-time went from 80.9 to 81.4, and StreamingBench real-time actually rose from 86.3 to the earlier table's 86.9. The past side is more striking. On the accuracy metric for revisiting past scenes (ASI, in the paper's notation), FIFO's 63.5 jumps to 71.6 under PHCM — 4.0 points higher even than the 67.6 of Full, which was holding the entire video. OVOBench's aggregate is also highest for PHCM (72.1 in the earlier table), topping both Full's 70.9 and FIFO's 70.1.
In fairness, on one other past-related metric (EPM, in the paper's notation) alone, PHCM's 62.6 falls just short of Full's 63.0. Notes don't beat holding the video on every past task.
On this foundation come the aggregate results. The 4-billion OneStreamer outperformed the nearly-three-times-larger 11-billion model (MOSS-VL-Realtime) on most evaluations. It's 25.0% higher on average than the base model, and 6.1% higher even against the strongest competing method on each benchmark.
| Category | Benchmark | OneStreamer-4B | vs. Qwen3-VL-4B | vs. MOSS-VL-Realtime (11B) |
|---|---|---|---|---|
| Perception & memory | OVOBench | 72.1 | +13.3 | +1.9 |
| StreamingBench Real-Time | 86.9 | +5.1 | +4.0 | |
| OVBench | 66.8 | +11.4 | +13.1 | |
| ODVBench | 71.3 | +13.7 | +7.4 | |
| Proactive response | ProactiveVQA | 48.7 | +14.4 | +1.5 |
| OmniMMI (with ASR) | 36.6 | +7.2 | +3.9 | |
| OVO-Timing (F1) | 41.6 | +12.2 | +3.1 | |
| ViSpeak | 2.87 | (base 2.41) | (11B 2.48) |
The way 'when to speak' was taught also paid off in numbers. Teaching only 27.5% of the state markers still beat both teaching every marker densely and a 'random selection matched for sheer quantity.' It learned better from less — the reading being that pruning common silence and keeping only the moments of speaking up was what did the trick.
This design's fate hangs entirely on 'the quality of the notes.' Because it carries the past as captions the model wrote rather than as video, when a caption is wrong, that error gets fossilized straight into memory. The source scene has already been pushed out of the window, so there's no way to rewind and check. A single mis-written line can contaminate every prediction that follows.
Why does this happen? In training, the model is taught with verified 'ground-truth captions,' but at run time there's no one alongside to verify. The model takes the note it just generated at face value and builds on top of it. From this gap between the training environment and the real environment, error accumulation sprouts. How to prevent errors in the memory it writes itself is a point to dig into further.
It also bears noting that the evaluation is entirely benchmark scores. Whether a 4B beating an 11B is a fit to these particular test sheets, or an edge that would hold up on real live streams beyond the test sheets, can't be settled by benchmarks alone. This isn't unique to this paper — it's a limit the whole field carries.
What happens if we move this principle onto a stream that isn't video — 'workplace collaboration'? Imagine an AI manager watching, from the sidelines, a team where meetings, chats, and work logs flow in nonstop all day. Even with a human PM on it, no one person can single-handedly watch every stream running all day. This is where the paper's design connects. What matters isn't the subject matter of video, but the very way of handling 'stream' as a data form.
Follow one scene through to the end. Tuesday afternoon, a client messages on chat: "Please push the delivery from Friday to next Tuesday." The team discusses briefly and decides to reorder the development sequence. Three days later, another team member asks, "Why did we change the delivery date back then?"
Until now, the standard solution for this kind of PM-style agent was to store every conversation and, when a question comes, search and pull up the relevant fragments. The OneStreamer-style picture has a different grain. In the moment the stream flows, the agent jots a line of its own: "Tue 15:00, at client's request delivery changed Fri→Tue, rationale as follows." For the question three days later, there's no need to trawl the entire log — just the recent context and that one note.
What ONTLE's AI PM handles is also not video but this kind of textual work stream. So the paper's technique can't be transplanted code-for-code. But the design principle of 'leaving a factual record of decisions while you watch' carries over as-is, beyond subject matter. Here, two instincts of a PM translate into data design. One: most moments are watched quietly, but the moments to speak up are rare — train only on silent cases and the agent will forever just read the room. Two: step in at the earliest point where the evidence becomes sufficient — the sense of timing that neither stumbles with a half-baked report nor reports only after it's all over.
And the weak link comes along too. What if the "rationale as follows" the agent wrote is wrong? Fortunately, unlike video, a work stream keeps the original log intact. That means you can slot in a verification step that cross-checks the note against the original log. The environment of text-based work actually compensates for the paper's weakness — the difficulty of preventing errors in the memory it writes itself.
No. Gemini and Seed were used only to create and verify the ground-truth captions for training. In actual service, OneStreamer writes its own notes alone. That's precisely why, at run time, there's no model alongside to verify, giving rise to the weakness of trusting and building on notes it wrote itself.
Text notes are far lighter than video fragments, so far more of them fit in the same room. Completed segments are coarsely bundled into sparse summaries, which together with the dense detailed records form a 'detail ↔ summary' hierarchy. That said, the paper only appends and accumulates both detailed captions and event summaries into the text history — it doesn't set a policy of erasing detailed records with summaries or capping the memory size.
The paper is CC BY 4.0 and a project homepage is public. That said, the training scale is large, so the reproduction difficulty is high. For the specific resource scale, see 'Training details' above.