Hand an AI a coding job or a terminal task and something odd happens. Same AI, yet on some days it breezes through, and on others it circles the same spot. Try to push its performance higher and you keep hitting the same wall. Did that last fix build real skill, or did it just memorize one trick that only works on this particular problem?
ModularRSI refuses to dodge that question. The core idea is this: if you want a signal for whether the skill an AI improved on its own is real, swap out the AI model that acts as its brain. A gain that was only a trick tends to fade along with the model; a gain that was real skill tends to hold. The paper doesn't prove that a model swap cleanly separates trick from skill in general — but on the models it evaluated, the reworked harness did carry over. This "portability" — the generalization that lets performance hold up even when you change the model or the task — is the heart of the paper. Let's trace how it got there.
First, one term. An agent harness is not the LLM — the core engine that answers questions in human language — itself. It is the "execution layer" that wraps that engine and actually gets work done. In what order should it think and act, which tool should it reach for and when, what should it distill from what it observes, how does it decide the job is finished — this entire "skeleton of how work gets done" is the harness.
A kitchen makes it easy. The LLM is the cook; the harness is the kitchen's workflow and tool layout. However brilliant the cook, if the workflow tangles and the knives are in the wrong drawer, the cooking slows down. An agent's real-world performance rides on the kitchen design as much as on the cook.
Recursive self-improvement (RSI) is the agent repairing this very kitchen on its own. It mines failures from its own execution logs, pins down "this approach was the problem," and rewrites the next workflow itself. The key is that the agent does it, not a person. Recent work has started to attempt this for real on long-horizon coding and terminal tasks.
The trouble is that this self-improvement easily fixes the wrong spot. The paper flags three traps.
| Trap in the usual approach | Why it gets confused with "real improvement" |
|---|---|
| Fixing the harness using the test problems (the benchmark) | Only that test's score rises; it may be a "trick" that doesn't carry over to unseen work |
| Fixing based on a single attempt | It muddles whether the flaw is in the skeleton itself or in the peculiar circumstances of that one job |
| Fixing the whole tangled harness at once | You can't tell which change was the one that helped, and unrelated features get snarled together |
All three traps converge on one thing: they blur where to fix and on what grounds. If you can't see the target, you can't tell whether you hit it.
ModularRSI pairs each trap with a remedy.
First, it fully separates where the practice problems and the exam come from. It collects 2,000 externally executable practice tasks, but keeps them from overlapping with the benchmarks used later to measure performance. If the practice set and the real exam share a source, a rising score tells you nothing about whether it's skill or leaked answers memorized. Pull the sources apart and that distinction becomes visible.
It also compares successful and failed attempts side by side, so it screens out not a one-off slip but an ingrained "bad habit." Looking at the difference between the two logs filters out "circumstances specific to this one task" and makes "flaws in the way of working itself" stand out. Failures with no successful counterpart are diagnosed on their own too, and even tasks that all succeeded get their efficiency tuned if there's room to finish faster. And when choosing which fixes to accept, it prioritizes fixes that are supported across multiple tasks. A fix that works in many places is more likely to be "skill" than a fix that fits only one task is to be a "trick."
And the biggest distinguishing feature is that it splits the skeleton into five parts and fixes them separately. Since fixing it all at once leaves you unable to tell what helped, the harness is divided into five modules.
Each module is reworked independently, only within its defined scope. Isolating the part being fixed makes it traceable which change produced which effect, and cuts down on unrelated features getting snarled in along the way. The separately polished modules then have their conflicts reconciled in a final integration step that merges them into a single harness.
The experiments ran on two test beds. SWE-Bench Verified is a standard benchmark for the coding ability to find and fix bugs in real software; TB2.0 measures the ability to compose commands in a terminal and get a job done.
The improved harness held onto its gains in all three directions. Laid out as a table of what was changed and what it proves:
| What was swapped | What it tests | What the result says |
|---|---|---|
| Unseen tasks in the same domain | Isn't this just memorized answers | Works on problems not in the practice set |
| Unseen tasks in a different domain | Isn't this a trick locked to one domain | Carries across domains |
| Swapping the base model (the cook) | Isn't this tuned to one model's quirks | Gains carried over to the other models the paper tried |
That last row is the heart of the paper. If an improvement were tuned only to one model's quirks, it would tend to fade the moment you swap in a different cook. Yet the kitchen ModularRSI reworked ran better even with a different cook standing at the station. On the models the paper evaluated, that points to what got fixed being not "this model, this test-taking trick" but "the way of working itself." A swap holding up on the models it tried isn't the same as proving, in general, that a model swap cleanly tells trick from skill — but within those models, the transfer showed up.
To be fair, there's something to flag. This result speaks to "how far it carries," not "how much" it improved. How many points the gain amounts to is something you'd have to read off the paper's own tables; this article alone can't tell you. Still, what self-improvement research has long wanted to prove was this generality, not the magnitude, to begin with. A large gain locked to one model and one test is a "trick"; a gain that holds up across the models and domains tested points toward "skill." What ModularRSI showed is the latter, on the models it evaluated.
This concern overlaps almost exactly with the daily life of the "AI project manager" we're building at ONTLE. An AI PM is a textbook long-horizon, multi-step agent. It calls various tools, distills what it observes, holds state within a narrow context, and judges for itself whether "this job is done." The five modules the paper carved out are, as it happens, our own worry list.
Let's follow one scene all the way through (this is still a design-stage assumption). Mid-task, a client asks to move the delivery date up. Say the AI PM reflects this change only in the schedule, declares "change applied" without touching the scope of deliverables already in progress. To hit the revised date it should have re-adjusted the scope too, but it didn't get that far.
First, resist the urge to tear up the whole prompt. Narrow the target. This mistake looks like a problem in the "task-completion judgment" module — the habit of declaring an end before all the changed conditions are reflected. But don't conclude from a single case. Following the paper's grain, start by digging through many clients' and many sessions' logs to see whether the same "premature close" shows up elsewhere. Once might be that one client's special circumstance; overlapping across many places makes the grounds for reworking that module far more solid. After the fix, gauge the effect by whether the same mistake dropped in other clients' work you didn't touch this time. Tuning our kitchen with our own test always looks like it works, so verification has to happen on work you didn't touch.
That said, don't just copy the paper wholesale. The five modules were carved to fit a coding agent, and an AI PM may need parts of a different grain. For instance, modules like "reading client intent" (the step of discerning the real need behind the words) or "generating interim reports" (the step of organizing progress in the client's language and sending it out) aren't in a coding benchmark but exist on our ground.
What the paper handed us is a hammer for coding. Our turn now is to take how it's held — splitting into modules, prioritizing fixes that overlap across many places, verifying on work you didn't touch — and forge a separate wrench for PM work.
This is exactly why the paper picked practice tasks that are "executable." Whether the code runs and the tests pass, whether a terminal command produced the intended result — a machine judges these objectively. Since success and failure split cleanly without a person grading each one, you can automatically extract "bad habits" from that difference.
The design itself is conservative precisely to reduce that risk. When choosing which fix to adopt it prioritizes the ones supported across multiple tasks, and it measures performance with an exam that doesn't overlap with practice, so there's little room to touch the wrong spot on the strength of a single lucky run. That said, the paper doesn't guarantee it prevents every regression — which is why a verification step, re-checking on work you didn't touch, remains at the end.
Not so. Put a better cook at the station and the same kitchen turns out better food. What the paper says isn't "all models are the same," but that once you've reworked the kitchen well, that improvement rode along even when the paper swapped in a different cook. A good model and a good harness aren't an either-or — they add up.