ProgramDistill is a benchmark that takes a new approach to evaluating coding agents. Instead of spelling out the exact behavior you want in an issue or a prompt the way most benchmarks do, it hands the agent a problem statement describing a feature's purpose and workflow, lets the agent poke at a finished, working reference application on its own, and asks it to implement the behavior it discovers there into an unfinished app. Without any human in the loop, the authors mined 1,975 verified behaviors across 26 apps and auto-generated 4,063 tasks, then ran their evaluations on subsets of that pool — three state-of-the-art models on a 12-app full-reconstruction suite, and nine coding agents on a 300-task partial-reconstruction subset (ProgramDistill-300).
Up to now, most ways of testing coding agents have looked like a homework worksheet. "Fix the bug where the login button doesn't respond," "Add a quantity stepper to the shopping cart" — the desired behavior is written out plainly in the issue or the instructions. The agent just reads the text and edits the code.
Real web development isn't nearly that considerate, though. In practice, instead of "please make this app behave like so," you often get a piece of software that already works well, shown to you with a "just like this one." The developer has to click through that app themselves, press the buttons, watch how the screen reacts, and reason their way to "ah, so this is what the feature does." Then they have to build that behavior into an app whose features aren't all filled in yet.
If existing benchmarks are "cook from a recipe," the situation this paper targets is closer to "you only get a one-line description of the dish, so you eat the finished version at a good restaurant and try to recreate the taste." There's no detailed recipe (no concrete behavior spec) — only a short statement of purpose and a finished product (a working app). That gap is exactly why an agent that scores well on existing benchmarks can't be assumed to do well in real web-development situations.
The heart of ProgramDistill is "slicing a working app into small pieces and making each piece's behavior replayable."
The authors factorize a finished application into features at varying levels of granularity — some large, some small. For each feature they attach a replayable behavior. That behavior is actually executed and reproduced when the ground-truth code change — the gold patch — is applied. Put simply, they've made it so a machine can automatically re-run and grade the statement "when this feature is implemented correctly, exactly this should happen on screen." The exam question and the grading criterion (the replay result) are bound together as one.
The name of the whole procedure the authors use to build the benchmark and evaluate agents on it is mine-craft-patch. The first two stages (mine, craft) are the human-free process of generating the benchmark, and the final stage (patch) is where the agent under evaluation actually solves the tasks.
| Stage | What it does (intuitive version) |
|---|---|
| mine | Digs replayable behaviors out of a working app |
| craft | Refines those behaviors into gradable tasks and builds the ground-truth code change (gold patch), verifying it by replay |
| patch | The agent under evaluation observes the reference app, implements the missing feature in the unfinished app, and is graded by replay verification |
With this pipeline the authors found 1,975 replay-verified behaviors across 26 applications and combined them into 4,063 tasks. The important part is that this whole process happened without human intervention. That means the benchmark can keep expanding without a person hand-labeling each answer.
Evaluation also splits two ways. One is full-application reconstruction, which deals with a cumulative workflow where you rebuild the entire app from scratch. The other is partial-application reconstruction, where you restore only the missing pieces of an app that's already largely filled in. In both cases the agent is given an auto-generated problem statement describing the feature's purpose and workflow, and it has to infer exactly how the feature should behave by interacting with the reference app. Here the authors add a tuning knob called restoration depth, which lets them finely control difficulty by how much has to be brought back. The distinguishing point is that this isn't a benchmark with fixed difficulty — it's one where you can dial the difficulty up and down.
The numbers in the abstract come from two experiments.
First, in the cumulative workflow of full-application reconstruction — three state-of-the-art models evaluated across 12 apps — GPT-6 Astra hit 49.2% and Claude Opus 5 hit 28.8% success. That's around half or below, which shows that looking at a working app and reassembling the whole thing is far from easy even for the latest models.
Next, in partial-application reconstruction — the nine-agent ProgramDistill-300 subset of 300 tasks — performance dropped sharply as restoration depth grew. Per the abstract, as restoration depth went from 1 to 8, success fell from 100% to 64.0%, and from 96% to 32%. When you only have to restore a piece or two shallowly, it's nearly perfect, but the deeper the stack of things you have to bring back, the faster it collapses. This is also empirical proof that the benchmark offers controlled difficulty.
Taken together, ProgramDistill matters in three ways. First, it's a large-scale benchmark that is scalable — it grows automatically without human intervention. Second, by adjusting difficulty you can diagnose an agent's weaknesses. Third, as the abstract explicitly notes, this difficulty-adjustable structure can serve as a natural foundation for curriculum-based training (teaching from easy to hard).
Judging from the abstract alone, a few honest limitations and open questions remain.
Beyond this, the abstract doesn't contain more specific methodological details or additional numbers, so I won't assert anything further. The right move is to check the original paper for the precise content.
ONTLE is a service that deploys AI project managers. That makes this paper directly relevant to us on two fronts.
First, "the ability to work only from a rough statement of purpose and infer the concrete behavior by looking at the finished product" overlaps exactly with the core competency of a working PM or agent. Real customers don't hand you a clean feature spec. They give you "make it like this service" or "just make this work," passing along a working example or a vague expectation. What ProgramDistill measures is precisely this situation: using a statement of purpose as the thread, reverse-inferring concrete behavior from a finished product, and implementing it on top of an incomplete state. The fact that even the latest models stall around the halfway mark on full reconstruction suggests that when you put agents into a real product, a "human PM's requirements-clarification step" is still essential.
Second, the idea of a verification system with adjustable difficulty is worth borrowing as is. The work ONTLE's agents handle also spans a wide range, from "shallow fixes" to "deep restorations where many pieces are tangled together." The result that performance plunged as restoration depth went from 1 to 8 reads as a practical lesson: rather than handing an agent a big chunk at once, it's safer to design the workflow so work is broken into small units and advanced cumulatively. And an automatic verification loop like replay verification — where "a machine re-checks that the result actually reproduces" — aligns with the goal of reducing the burden of a human eyeballing every agent output.
From a RAG or agent-pipeline standpoint, too, the paper shows that "spec text" isn't the whole of context — interacting with a working reference is itself a powerful input. It pushes you to consider designs that, beyond retrieving documents to stuff into context, add a step of actually observing and replaying behavior to confirm it, right in the pipeline.
Existing benchmarks write the concrete behavior into a text spec — an issue or instructions — and have you implement it as written. ProgramDistill differs in that it gives you a problem statement describing the feature's purpose and workflow, but requires you to infer the concrete behavior yourself by interacting with a working reference app and implement it in an unfinished app.
The mine and craft stages of the mine-craft-patch pipeline decompose an app into features and replay-verify each behavior with a ground-truth code change (gold patch). According to the abstract, this process generated 1,975 verified behaviors and 4,063 tasks across 26 apps without human intervention.
Per the abstract, in the cumulative workflow of full-application reconstruction (three models across 12 apps), GPT-6 Astra showed 49.2% and Claude Opus 5 showed 28.8% success. In partial reconstruction on the nine-agent ProgramDistill-300 subset, as restoration depth grew from 1 to 8, success fell from 100% to 64.0%, and from 96% to 32%.
The abstract notes that this difficulty-adjustable structure can serve as a natural foundation for curriculum-based training that teaches from easy to hard. That said, the specific training methods are outside the scope of the abstract.