GraphForge is a method for synthesizing training data for "working agents" — agents that read documents, operate tools, and produce deliverables — by building tasks and their grading rubrics on top of real work files. The core idea is to weave the relationships between files into an evidence graph, then compile the task description and the evaluation criteria out of that graph at the same time. Fine-tuning Qwen3.6-27B on the resulting 2,169 training trajectories raised its scores across all three benchmarks that measure on-the-job capability.
Teaching a working agent requires two things at once. One is realistic work: an actual workspace where reports, spreadsheets, and policy documents are all jumbled together. The other is a scoring sheet that can tell whether the job was actually done well. It's the same as handing a new hire a realistic assignment versus handing them the standard by which their output will be judged — two separate preparations.
Existing approaches to synthesizing training data have given up one or the other. The two pipelines the paper uses for comparison each represent one of those two camps.
| Approach | Work environment | Verification | Scale |
|---|---|---|---|
| EnvCraft | Model-generated workspace | State-check scripts | 20K tasks |
| NexForge | Real files | None | 5.6K tasks |
| GraphForge | Real files | Evidence-grounded agent judge | 2.1K tasks |
EnvCraft stamps out the files themselves with a model, which makes them less realistic and less diverse. On top of that, because its verification uses Python scripts that only inspect the "state" of the workspace, it can't read file contents and misses errors buried inside document deliverables. NexForge goes the other way: it builds tasks on top of real files but provides no per-task rubric or verifier, so there's no way to assess the quality of what the agent produces. The takeaway is that building diverse tasks from real files while also attaching trustworthy grading criteria to them has been that hard.
Where GraphForge gets clever is in separating the source of "direction" from the source of "content." Ask a model to invent tasks on its own and it gravitates toward common occupations and predictable analysis tasks. So GraphForge nails down the direction of a task with occupational data, and draws all of the specific content and verification from the real files. The whole process runs in five stages.
First, seeds are drawn from an occupational database called O*NET. It's an official occupational dictionary in the U.S. Department of Labor lineage, cataloging the tasks and detailed work activities (DWAs) for each occupation. Keeping only the tasks that can be performed digitally leaves 246 occupations across 16 industries, 891 work types, and 3,419 valid combinations. To avoid clustering, seeds that don't overlap with ones already chosen are picked first, spreading the diversity wide.
The seed itself contains no company name, no incident, no data. It's an empty template holding only "who handles what kind of demand, and chiefly how." In the next stage, a search agent finds a real, publicly documented case and gathers the files that work needs, assembling a workspace. At this point every file gets an underlying role that isn't visible on the surface. Core files drive the calculations and judgments; supporting files provide context; confuser files are plausible-looking but ultimately useless traps; and peripheral files add realistic noise. This role sheet is never shown to the agent solving the task.
On top of that, an evidence graph is built. The graph's nodes are facts or values that a given file provides, and the edges are the cross-file dependencies — connections that have to be compared, contrasted, or derived. It's from this graph that the task description and the grading criteria are compiled together. Each rubric item is bound to the files it needs for evaluation as evidence anchors, along with the procedure for how to verify it. The agent solving the task sees only the task statement and requirements; the judge receives the anchors and the verification procedure. In effect, the scoring sheet is nailed to the files, but the location of those nails never leaks onto the question sheet.
It doesn't stop there. Reading the task on paper alone won't catch all the ambiguity of a long assignment, so a strong teacher model actually solves the task once. A revision agent then cross-checks that run against the original files and graph — is the task natural, are the requested values actually backed by the files, is every item properly anchored — and returns only the parts that need fixing. Finally, after a mechanical check that the deliverable was actually produced, an agent judge scores it item by item, moving back and forth between the original files and the output. Abnormal trajectories, such as pointlessly repeated tool calls, are filtered out. Only what passes this gauntlet makes it into the training data.
The authors fine-tuned Qwen3.6-27B on GraphForge data and measured it on three benchmarks. The numbers come from different evaluation harnesses (agent-execution frameworks), so they have to be read with their conditions attached.
| Benchmark | Result | Improvement | Evaluation harness |
|---|---|---|---|
| GDPVal (Elo) | 1445.7 | +65.7 | OpenHands |
| Workspace-Bench-Lite | 63.7 | +7.7 | Claude Code |
| SpreadsheetBench II | 24.0 | +13.7 | Claude Code |
The GDPVal score is an Elo rating, not a percentage, and its meaning needs to be read precisely. This 1445.7 is not a direct comparison against human work. It's an internal Elo the authors produced by pitting various model-and-harness combinations against one another, anchoring the scale by fixing the GLM-5.3 (OpenHands) combination at 1667; 1445.7 is a relative position on that scale. GDPVal itself is a benchmark that evaluates 1,320 tasks across 44 occupations against expert rubrics, and the authors used a 220-task gold subset of it. What stands out is the jump on the spreadsheet side. The 24.0 is the figure after adding +13.7, which means the starting point was somewhere around 10.3 — the biggest relative leap of the three.
On top of this, the authors ran rejection fine-tuning, in which the fine-tuned model's own execution outputs are filtered by the evidence-grounded rubric and only the good ones are used to train it again. The fact that scores rose further on all three benchmarks suggests the rubric is more than just an evaluation tool — it also acts as a signal for selecting "which outputs are actually usable." And the fact that the same data worked on a second base model with a different architecture (Qwen3.6-35B-A3B) hints that this isn't a trick that only fits one particular model.
The scale invites another read. GraphForge's task count is 2.1K, far fewer than the comparison pipelines (20K, 5.6K). If fewer tasks produced better results, it points toward verified-but-few data beating unverified-but-many. The data and models are public.
Honestly, there are a few things worth flagging.
The final grading is done by an agent judge (LLM-based), not a human. Anchoring the rubric to the files should reduce ungrounded verdicts, but the reliability of the judge itself remains an open question. The paper's own phrasing is cautious rather than assertive — something like "the rubric appears to provide a useful selection signal."
The fact that the training trajectories come from a strong teacher model's runs is a double-edged sword. Task types the teacher can't solve are unlikely to end up in the data either, and the teacher's habits and limits may be inherited. The authors are also explicit that the evidence graph is "an intermediate representation, not ground truth." The claims in the graph are merely a scaffold that must remain re-verifiable against the original files, so quality hinges on how faithfully the graph reflects those files.
The numbers above were measured on specific benchmarks under specific harnesses. How far they carry over to other tasks or other execution environments is beyond what's reported. The paper is said to have a separate section discussing limitations, but the excerpt this piece was given doesn't include those details, so the specifics aren't asserted here.
Picture the moment an AI PM at ONTLE reports a task as "complete." A customer has uploaded ecount settlement records, a supplier's quote spreadsheet, and a contract-terms PDF, all mixed into one folder, and the PM claims to have read them and produced the monthly settlement report. The question is what comes next: was it actually done right? The "ghost-done" problem ONTLE has been burned by repeatedly — where the status flips to complete but the actual result is empty or wrong — arises at exactly this point. Verification so far tends to stay at the level of a human eyeballing "does the deliverable look plausible."
GraphForge's evidence-anchor idea changes precisely this scene. Instead of vaguely judging "the settlement total is correct," the files and cells needed for that judgment are nailed into the rubric items ahead of time. Column D's total in the quote spreadsheet, the commission clause in the contract PDF, the relevant voucher in the ecount records — which value in which file each pass condition must be checked against is decided at the very moment the task is created. Then the PM's "complete" turns from a feeling into traceable grounds: "this item passed by cross-checking against this value in this file." The structure of hiding the anchors from the PM solving the task and giving them only to the judge is just as useful here. It forces the PM to actually read the files and work, rather than reverse-engineering and mimicking the known answer path. This isn't a call for ONTLE to adopt this pipeline tomorrow — the point is that the design principle of "binding completion to source files and verifying against them" is closer to a head-on fix for the ghost-done problem.
The paper states that it has released both the synthetic data and the trained models. It says they can be accessed from the groundhogLLM/graphforge collection on HuggingFace. That said, the license and the actual conditions for reproduction should be confirmed directly on the public pages.
It's an official occupational database that catalogs the tasks and detailed activities performed in each occupation. When a model invents tasks freely, it skews toward common occupations and predictable analysis; using occupational data as the seed lets you keep the direction of the tasks spread wide while fixing it in place. GraphForge keeps only the tasks that can be performed digitally.
No. The paper is explicit that the evidence graph is an intermediate representation, not ground truth. The claims it holds must always be re-verifiable against the original files, and the final grounding is the real files, full stop.
If ordinary fine-tuning (SFT) is the stage where the model learns the teacher model's execution trajectories as-is, rejection fine-tuning is where the trained model produces its own outputs on new tasks and then learns again only from the ones the rubric judges to be good. In this paper the selection criterion was the evidence-grounded rubric, and it yielded further gains across all three benchmarks.