Whether an autonomous coding agent actually gets good work done depends not only on how smart the underlying model is, but on how well the "harness"—the execution skeleton that wraps the model—is designed. This paper takes harnesses, which until now have only ever been evaluated as a whole, and decomposes them into three parts—planning, action space, and context management—then swaps them in and out one at a time across 176 matched conditions. The headline contribution is that it reveals, part by part, "what works and when."
No matter how skilled the cook (the model) is, if the kitchen's (the harness's) workflow and tool layout are a mess, a good dish is hard to produce. The same goes for coding agents. Even with the same language model, performance on long-horizon software tasks changes dramatically depending on how you make the model plan, what tools you hand it, and how you manage the conversation history.
The trouble is that prior work has evaluated the harness as a single monolithic system. If you only compare Kitchen A against Kitchen B as wholes, you can't tell whether it was the workflow, the knife, or the fridge that actually boosted performance. Which individual part is really doing the work stayed blurry.
So this paper flips the approach. It builds a lightweight harness with a fixed execution loop (the observe → decide → act skeleton the agent repeats), and then experiments by varying only these three parts:
The researchers ran four models on two kinds of benchmarks (SWE-Bench Verified, Terminal-Bench 2.1). Think of the first as fixing real software issues, and the second as terminal- and command-line-centric tasks. They then compared 176 configurations with conditions carefully "matched." Only by holding everything else constant and changing a single part can you attribute a performance difference to that part.
Here are the three parts in everyday terms.
Context management — how you keep your desk tidy. As a task drags on, paperwork (conversations, logs, code) piles up on the desk. The desk (the context window—the amount of information the model can see at once) has a limit, and once it overflows, the whole task collapses. The researchers compared five tidying strategies, two of which stand in contrast. One is rule-based deletion (elision—cutting less important content by fixed rules); the other is LLM-based summarization (letting the model shrink things on its own). Another variable is whether to make the cut content recoverable so it can be pulled back later.
Planning — a note before you start. It's the habit of jotting down the order of operations before beginning. For a novice it's a scaffold that keeps you from getting lost; for an expert it's a path already known, so it doesn't help much, but it still cuts down on wasted steps.
Action space — how the toolbox is assembled. You can hand the agent several well-defined, specialized tools (predefined tools—purpose-built functions like file editing and search), or conversely give it a single shell command (bash) and let it figure things out. It's the difference between one all-purpose Swiss Army knife and a set of task-specific tools.
The abstract lays out four findings. Since the abstract gives no concrete numbers, I'll convey the direction of each.
| Part | Key finding |
|---|---|
| Context management | The value grows as the context-window budget gets tight. Most of the gain comes from preventing failures caused by context overflow. |
| Context strategy | Staging rule-based deletion ahead of LLM summarization gives the best overall efficiency. Machinery that makes cut content recoverable yields no accuracy gain, because the model barely uses it. |
| Planning | For weak models it acts as a scaffold that props up accuracy; for strong models its role shifts to saving cost. The change in accuracy is small. |
| Action space | Models with low bash proficiency perform better with predefined tools. Conversely, models fluent in bash work fine with just the shell, and dramatically lower cost especially on command-line-centric tasks. |
The researchers explain these effects through trajectory-level analysis (a trajectory being the agent's entire flow of actions while solving a task). In short, the three parts act on different places.
This lens matters because it distinguishes not whether each part "improves performance" but how it intervenes. That gives you a basis for picking and matching the harness to the model's strength and the budget situation.
There are points worth stating plainly.
First, the experiments are limited to four models and two kinds of benchmarks (SWE-Bench Verified, Terminal-Bench 2.1). Whether the same conclusions hold for other model families or tasks of a different character is beyond this paper's scope.
Second, the researchers deliberately used a lightweight harness with a fixed execution loop. That's a choice made for controlled comparison, but in production harnesses where the execution loop itself is sophisticated, the interactions between parts could play out differently.
Third, the finding that "the model barely uses the recoverable-deletion machinery" is intriguing, but whether this stems from current models' habits or from an awkward design is hard to pin down from the abstract alone. It's a point that will need revisiting as models change.
ONTLE is a service that dispatches AI project managers. What we deal with is, in the end, the same problem: with what skeleton do you wrap an agent that works over long stretches. Here's what we'd carry from this paper into practice.
The tighter the budget, context management first. In an operating environment where you can't use the context window lavishly, the value of a tidying strategy becomes especially large. And the fact that much of that gain comes from preventing "overflow-into-total-failure" accidents ties directly to real-world service stability. The finding that it was more efficient to run cheap rule-based deletion first and layer LLM summarization on top is worth following as-is.
Vary the role of planning by model generation. For a weak model, planning is an accuracy scaffold; for a strong model, it's closer to a cost-saving measure than an accuracy boost. It makes sense to tune "how strongly to enforce planning" depending on which model you attach.
More tools is not automatically the answer. A model fluent in bash runs fine on a shell-centric interface alone, and on command-line-centric work it actually cuts cost substantially. Conversely, a model with low shell proficiency is propped up by predefined tools. Harness design should move toward being model- and budget-aware—which is both this paper's conclusion and something that aligns with our product design philosophy.
Above all, the biggest gift this research offers is a modular evaluation framework that shifts the lens from monolithic comparison to part-by-part comparison. From now on, each time a new harness part appears, we have a shared language for asking "where, and why, does this act?"
It's the execution skeleton that wraps a language model so it can be put to work on real coding tasks. It refers to the whole apparatus—making the model plan, handing it tools, managing a work history that keeps growing—that turns the model's capability into long-horizon software performance.
The point of this paper is that you can't single one out. Context management matters more the tighter the budget, planning matters more the weaker the model, and predefined tools matter more the lower a model's shell proficiency. The answer shifts with the situation.
According to the abstract, even when the recoverability machinery was added, the model barely used it, and as a result there was no accuracy gain. It only added the complexity of an extra mechanism.
Caution is needed. The experiments are limited to four models and two kinds of benchmarks, and the results come from a lightweight harness with a fixed execution loop. Take the direction as a reference, but re-validation is recommended in other environments.