Say episode 1 has a scene where the gang's eldest brother is called "형님." The English subtitles render it as "boss." But in episode 12 it becomes "big bro," and in episode 40 it's "brother." All three are technically correct. Yet viewers still trip over it: "Wait, why does this guy's name keep changing?" This slow slippage of a translation over the course of a series is what we call drift.
Pull any single line out on its own and none of the three is wrong. The problem lives outside the sentence, in the consistency that runs across the entire series. SMART is a subtitle translation system aimed at exactly this spot. It is a setup where several AIs with divided roles collaborate (a multi-agent system), and the key is that it fixes its own working process while the translation is still underway. On Subtitle Arena, the evaluation environment the authors released alongside it, it took first place in all 15 translation directions.
Put simply, the bottleneck SMART points to is not the translation model's skill. It's a structural mismatch: easy lines get burdened with overkill processing, while hard lines never get the specialist they actually need. Why this happens becomes concrete in three limitations of the existing approaches.
First, the ability to hold onto long context all the way through and keep things consistent is weak. There are approaches that tune a single AI for subtitles, or feed it surrounding lines and a plot summary, but in the end they think at the sentence level. So recurring terms of address and proper nouns drift episode by episode, just like "형님" above.
Second, the collaboration pipeline is fixed regardless of scene difficulty. There are also approaches where several AIs split up translation, review, and revision, but the roles and procedures are nailed down in advance. A plain greeting or a culturally loaded joke, it all rides the same pipeline unchanged.
Third, they can't pull in and reflect production context like genre, period setting, and culture on the fly.
On top of this comes one more physical constraint unique to subtitles: on-screen display rules like the number of characters that fit on a line, and the amount of text relative to how long the subtitle stays on screen (reading speed). Getting the meaning right isn't enough. If the translation runs long and spills off the screen, that alone is a failure.
SMART's core idea fits in one sentence. Without retraining the underlying large language model (an AI that generates text, trained on vast amounts of writing, an LLM), it fixes its own working process while the translation is underway. It doesn't touch the model's internal numbers (its parameters). It only rewrites, in natural language, the instructions given to the model and the rules for "which sentence goes to whom." Tuning the configuration at the moment the service is actually running like this is called test-time training, that is, learning done while doing the real work.
SMART evolves through two kinds of activity. One is doing the translation (the translation loop), and the other is improving the working process itself based on those results (the improvement loop).
The device sitting at the very bottom is a single notebook that the whole translation team shares. The authors call it persistent series memory. Four things accumulate in this notebook: a terminology pool, character profiles, scene and domain knowledge, and an idiom bank. Before translating an episode, the system pulls out and reads the notebook so far, and after the translation is done it merges any newly confirmed information back into the notebook. Every translator and tool consults this notebook. So once "형님" is locked in as "boss," the translator on episode 40 doesn't agonize over it afresh each time; they just pull "last time it was boss" straight from the notebook. Consistency climbs from the sentence level up to the series level.
Why keep a separate notebook? Cramming all the information into the prompt (the bundle of instructions handed to the model in one shot) every time causes three kinds of loss. There's a limit to how much you can feed in at once (the context window), so you can't fit an entire long series. Even if you could, the model would easily lose track of the key points it should focus on. And the more characters, the higher the cost. So rather than carrying the whole thing, it's better to pull out only the pieces this particular sentence needs. The notebook is essentially a device that stores long-term memory, which exceeds any individual call's context window, separately on the outside.
Alongside this, it keeps a lightweight content profile holding the series' genre, setting, and main characters. For ambiguous slang or cultural references, it fetches the facts via external search, but then interprets those facts in the context of the current scene before storing them. The idea is to store the external knowledge's "facts" and the subtitle-appropriate "tone and wording" separately.
Above that, a router divides up the work. Routing is the job of deciding which sentence goes to which translator. Not every translator is summoned for every sentence. Looking at signals like sentence length, the scene's tone, and slang or profanity, it picks which translators to call. The translators handle different strengths. Some specialize in semantic fidelity, some in naturalness, and others in controlling subtitle length or handling colloquial speech. Easy sentences get fewer translators; tricky sentences get more specialists.
The selected translators each put forward candidate translations. At this point they pull from the notebook the confirmed terminology and past sentences translated similarly, and they call tools such as subtitle-constraint checks and target-language fluency checks.
Once the candidates are gathered, an AI playing the role of judge scores each candidate on five axes: semantic accuracy, fluency, style and tone, long-range consistency, and subtitle display constraints. Along with the scores, it produces a critique pointing out "what falls short and why." It picks the highest-scoring candidate, polishes it once more with this critique, and reflects the confirmed translation into the notebook. When the episode ends, it sweeps the whole thing once more to fix residual consistency errors that are hard to catch from any single sentence alone.
There's a blind spot worth flagging right here: the judge and the translators run on the same underlying model. An error that model can't see in the first place, the judge can't see either. Because it's a setup where it grades its own answers, there remains a risk that it can't filter out the model's own biases on its own. The authors' testing with a weaker model and a different evaluator reads as a response to this very worry.
The evolution climbs one layer higher. Each time a training batch finishes, the system gathers the candidate scores and critiques and fixes two things: the instructions for underperforming translators (correcting repeated failures), and the routing rules (shifting allocation toward the higher-scoring configurations). Both are rewritten in natural language, but it's constrained rewriting, not free-for-all tearing apart. For instructions, it only touches the underperforming translators and leaves the roles and flow intact. For routing, it keeps floor conditions like "at least 2 translators per category" and only rewrites the allocation table. These are safeguards to keep the self-correction from running wild. And it's applied not sentence by sentence, but just once per training cycle (an epoch, the unit of one pass through the data).
One final design choice governs the credibility of the evaluation. The series is split into two batches. On the front batch, it actually translates some sentences and runs the evolution above, then "freezes" the resulting configuration and applies it to the remaining episodes. The reason is simple: using the back-half results, which are what you're supposed to be evaluating, to further tweak the configuration would be like secretly studying from the exam questions, inflating the score. The one exception: the notebook alone keeps growing even during the application phase.
To sum up, where SMART parts ways with other self-improvement approaches is not "what it changes" but "when, and while preserving what, it changes." Within a single long translation task, it preserves the terminology and style already built up while improving the next sentence.
To measure performance, you first need a yardstick, and existing subtitle datasets couldn't measure series consistency. Because the sentences were scattered individually, there was simply no way to measure whether the same term of address held up even at episode 40. So the authors built the evaluation environment from scratch.
Subtitle Arena starts from the public subtitle collection OpenSubtitles2024, but with one decisive difference: instead of laying sentences out in a row, it regroups them by episode and by series. Using the movie database identifier (IMDb) and the season and episode numbers, it reconstructs the correspondence between the two languages, preserves the timecodes while correcting format errors, discards misaligned pairs, and bundles the same series together to keep the long narrative structure alive. The scale: 192 series, 14 genres, and 15 target language locales. English originals numbering 6,267 were rendered into 15 locales, for 70,664 bilingual episode pairs in total — but that's the count of "pairs," where the same episode is tallied separately per language, not the number of episodes. Production years span from 1959 to 2023, and episodes per series range from 2 to 198. By locale, Chinese is the largest at 6,267 pairs, and Korean holds 3,243 pairs.
The scorer is SubMQM. It's an adaptation for subtitles of MQM (Multidimensional Quality Metrics), the international scale that measures translation quality by deducting points for errors, where an AI evaluator assigns a penalty for each error across 7 dimensions. Lower scores are better. For the dimension weights, accuracy carries the largest at 0.30, terminology and fluency get 0.20 each, and the rest get smaller shares. In other words, whether the meaning is right is weighted most heavily.
There are three groups of comparison targets: community subtitles (treated as not ground truth, since their quality is uneven), several kinds of single-call AI, and the role-divided agent approaches TransAgent and DRT. For fairness, both agent comparison groups were reimplemented on the same commercial model as SMART.
That said, SubMQM is an automatic evaluation that uses AI as the judge. Applying the same evaluator to every system secures fairness in relative comparison, but the possibility of the evaluator's own bias remains an inherent limit of automatic metrics. You have to read the numbers with this premise in mind.
The clearest signal is that putting more specialists on hard lines — the dynamic allocation — actually worked. On top of the first place across all 15 directions mentioned earlier, it cut the average penalty by 6.9% versus the strongest agent comparison group. On a separate public benchmark (MuSC), it was best in all 4 language pairs, and in a human-rated overall evaluation it scored the highest at 4.50 out of 5.
Looking at a representative direction involving Korean reveals the texture of the numbers. Going from English to Korean, SMART's overall penalty is 0.92, lower than TransAgent's 0.97 and DRT's 0.99 in the same direction. Conversely, going from Korean to English it's 0.47, lower than TransAgent's 0.52 and DRT's 0.57. Since lower penalties are better, it leads in both directions.
That doesn't mean it's first in every sub-dimension. Going from English to Italian, on technical dimensions like character count and line breaks, SMART comes in at 2.37, actually higher than some of the comparison groups. Because the overall score is weighted toward semantic fidelity, it reads as a structure where a loss on one dimension is made up on another to carry the overall lead. The "cut by 6.9%" figure, too, is an average comparison against the strongest agent system, not a clean sweep across every metric and every opponent. This needs to be read separately.
Still, it clearly demonstrates one thing: even without retraining the model, evolving only the instructions and allocation rules in natural language can beat fixed-pipeline agents on the consistency problem of long-form content. That's as far as the evidence supports. A generalization like "operational design beats fine-tuning" can't be claimed from this result alone.
There's no free lunch. SMART generates multiple candidates per sentence, has the judge score them by axis, polishes once more with a critique, and evolves the configuration on the front batch. Every one of these steps is an additional call. Compared to a translation that finishes in a single call, the number of calls multiplies several times over, and cost and response latency rise accordingly. The concrete figures are covered in the paper's appendix, and I won't invent numbers that aren't there. Pointing only at the direction: for long-form content where consistency pays off, it's worth it; for short, standalone tasks, it's overkill.
One assumption is also baked in: you need material to evolve on. Because SMART evolves its configuration on the front batch, it's favorable for series with episodes long enough, but short content simply has little data to evolve on. The train-apply separation that guards against overfitting simultaneously imposes the condition that "there must be material to evolve on."
Translation and project management look far apart, but the problem at the bottom is the same: reusing past decisions consistently, even further down the line.
Let's go to a concrete scene. A client contact sends a message to the AI project manager: "Please push next Friday's deadline out to the following week." The manager has to write a reply. But three months ago, with this same client, they'd agreed that "deadline changes are reflected only after a new purchase order is reissued to confirm them." A manager who judges from a blank slate every time would miss this agreement and reply, "Sure, we'll adjust it." Later, when the schedule slips without a purchase order, accountability gets tangled.
Just as SMART solved long-haul consistency in translation with external memory, project management, too, raises its odds of success when it systematically accumulates the context, terminology, and agreements of past communications and pulls them back out — instead of judging afresh every time. In the scene above, if the manager pulls that agreement from the notebook and replies, "The change is confirmed after the purchase order is reissued," then they've consistently reused the decision from the third request on the tenth.
The review stage is worth carrying over, too. Before sending a reply to the client, a role-separated reviewer runs a self-check on axes like terminology consistency, the client's tone, and whether the request is satisfied. When a repeated failure shows up, leave the roles intact and fix only the instructions, in a constrained way. But bringing it over as-is comes with conditions. Enough early interactions have to accumulate to serve as training material, and the blind spot of the reviewer being the same model as the reply generator has to be filled in by cross-verification. The fact that translation's shared notebook and project memory look alike doesn't make the two the same system. What you bring over is the principle, not the feature.
The device that reduces that risk sits at the moment of writing to the notebook. A translation is reflected into the notebook only after the judge scores it on five axes and it's "confirmed" with one round of polishing from the critique. A raw candidate isn't stored directly. On top of that, when the episode ends, the whole thing is swept again to fix residual consistency errors. Still, an error the judge fails to catch can propagate downstream — and that's the spot that touches the blind spot of the judge and the translators being the same model.
Because it's not several physically distinct models, but a single underlying model dressed in different roles and instructions to divide the labor. It's like the same person swapping between a translator hat, a judge hat, and a dispatcher hat depending on the situation. Dividing the roles lets you separate "the side that produces candidates" from "the side that grades them," but since the underlying head is the same, the shared bias remains.
Order matters. SMART evolves its configuration on the front batch, then freezes it and applies it to the back, and the notebook keeps accumulating as the episodes go. Since terminology and terms of address confirmed in earlier episodes become the material for translating later ones, keeping to the flow in which context accumulates fits the design's intent.