When a language model learns information sequentially over 100 rounds as it arrives over time, later learning erases earlier learning, a phenomenon known as "catastrophic forgetting." To tackle this head-on, this paper defines a new setting called "long-horizon memorization" and shows, through systematic experiments, that combining several continual learning techniques that each target a different cause of forgetting leads to far better memory than any single technique alone. The key combination lifted the final average retention from 1.2% for plain sequential fine-tuning to 34.9%, roughly a 28x improvement.
Language models often aren't trained once and done. They frequently operate in settings where new information keeps arriving. You stack today's new facts on top of what you learned yesterday, and tomorrow you stack more. The trouble is that repeating this process causes the model to forget what it learned before. This is what we call "catastrophic forgetting."
Here's an analogy. Suppose you overwrite a single whiteboard with new notes every day, and you never save what you wrote earlier anywhere else. The first few days are fine, but after roughly 100 overwrites, almost nothing of what you first wrote remains. A model's weights (the numerical values that hold what the model has learned) behave similarly. As you keep adjusting the values to learn a new task, the values you had tuned for earlier tasks get pushed out.
This paper tests the problem under especially harsh conditions. In the "long-horizon memorization" setting the authors define, a model learns 100 query-answer tasks through continual supervised fine-tuning. Two constraints apply. First, you do not save the original training examples you saw earlier and replay them later (no storing or reusing the original past data). Second, at inference time you are not given a task identifier telling the model "this is task number such-and-such." In other words, the model has to tell apart and remember 100 tasks on its own, without any hints.
When the authors tried a range of existing continual learning techniques one by one, none could sustain strong memory over a horizon this long. So they shifted their thinking: "If no single technique works, what if we combine techniques that each target a different cause?"
The paper's hypothesis is this: forgetting has different sources, and techniques that target different sources are complementary, so combining them should be more effective. The authors organize this combination along two design dimensions.
The first dimension is "what to preserve." Each time you train on something new, you decide how to hold on to earlier information. This is the "anchor." There are three kinds.
The second dimension is "where to store it." The authors use a low-rank allocation rule. This is where LoRA (Low-Rank Adaptation) comes in. LoRA is a method that learns only small "add-on modules" to store changes, instead of overhauling the entire model. Rather than painting directly on a large picture, it's like laying a thin transparent film over it, one at a time, to make edits. The original stays less damaged, and the edits live on the film.
The "low-rank allocation rule" decides how to arrange and manage these films: where to put the changes for each new task, and how to overlay the films from multiple tasks. The paper's best-performing combination uses "merged LoRA" here, a scheme that merges and manages the LoRA modules learned across multiple tasks as one.
To summarize, the anchor handles "what to preserve," and low-rank allocation handles "where to keep it." Multiply the two and you get quite a few combinations. The catch is that the number of combinations explodes.
To search this vast design space, the authors use two experimental methods.
To properly test this hypothesis, the authors also built three distinct 100-task memorization datasets themselves. This design rules out the chance that something only works on one dataset.
The key result stated in the abstract is clear.
| Item | Value stated in the abstract |
|---|---|
| Final average retention for plain sequential fine-tuning | 1.2% |
| Final average retention for the best combination | 34.9% |
| Improvement factor | About 28x |
| Ranking of the best combination | Top 3 on all three datasets |
The best-performing method uses all three kinds of anchors (data, function, weight) and combines them with merged LoRA. This combination landed in the top 3 on all three datasets, showing it isn't a fluke that only works on one particular dataset.
A particularly interesting finding concerns "which technique contributed the most." According to the abstract, the data anchor and merged LoRA gave the largest average gains, and these two interacted "super-additively" across all three datasets. Super-additive means that using the two together produces a larger effect than simply summing their individual effects. One plus one adds up to more than two. This is precisely the empirical evidence that backs the paper's central claim: combining complementary techniques produces synergy.
To sum up the significance: on the long-horizon memorization problem, carefully combining techniques that each target a different cause of forgetting goes much further than hunting for "one single magic technique."
There are points worth noting honestly.
First, 34.9% is clearly a dramatic improvement over the plain approach (1.2%), but in absolute terms it still means forgetting a substantial share of the 100 tasks. Long-horizon memorization has been greatly improved, not "solved."
Second, this setting deals with a particular form of memory: 100 query-answer tasks. From the abstract alone we can't know the specific nature of each task, the scale of the datasets, or how similar the tasks are to one another. Whether it transfers directly to the far more varied and intermixed information streams you meet in real products needs separate validation.
Third, the best combination uses all three anchors plus merged LoRA, so it has many moving parts. How large the implementation, tuning, and compute costs are as a result, and whether the advantage holds as the number of tasks keeps growing past 100, remain open questions. (Since the abstract doesn't give these specific numbers, we won't assert anything here.)
ONTLE is a service that deploys AI project managers. An AI PM is, by nature, something that "has to keep learning new things over time." It can only do a proper manager's job if it accumulates yesterday's meeting decisions, last week's customer requirements, and the context of a project from a month ago without forgetting them. The "long-horizon memorization" this paper addresses is exactly the core challenge of that ability.
Here are a few implications this research offers for real-world products.
That said, as the 34.9% result reminds us, perfect memory is still a distant goal. In practice, rather than relying on the model's internal memory alone, a realistic approach is to pair it with verifiable external records (documents, databases) as a safety net so that "even if it forgets, it can look it up again."
It's the phenomenon where, as a language model adjusts its internal values (weights) to learn a new task, the values it had tuned for earlier tasks get pushed out, so it forgets what it learned before. It's similar to how the first notes disappear when you keep overwriting a whiteboard.
According to the abstract, plain sequential fine-tuning has a final average retention of 1.2%, while the best combination of three anchors plus merged LoRA scored 34.9%, about a 28x improvement. Note that the absolute value of 34.9% still means a substantial share is forgotten, so the problem isn't fully solved.
Anchors (data, function, weight) decide "what to preserve." Here, the data anchor isn't about storing the originals; it's generative replay, where the most recent model generates synthetic data carrying past information and mixes it into new training. The low-rank allocation rule and merged LoRA decide "where to keep the changes." The paper showed that when you combine these two dimensions, the data anchor and merged LoRA in particular reinforce each other super-additively.
The research setting deals with a particular form of memory, 100 query-answer tasks, so applying it directly to the varied information streams of a real product would need additional validation. That said, the principle "combine complementary memory techniques" and the direction "design it to hold up without hoarding originals, regenerating past information on its own (generative replay) and coping without identifiers" are useful guidance for designing real-world agents.