This paper proposes a way to measure the quality of the "fork-in-the-road" choices that LLM agents make partway through long-horizon tasks (work that unfolds over many steps and takes a while). The authors call this capacity for good long-term decision-making the agent's "taste," and they evaluate it using Taste-Bench, a benchmark built almost entirely automatically, with little human involvement. And here's the surprising part: even the best-performing model got only 59.7% of these problems right, yet the authors show that taste can be improved through training.
Today's AI agents no longer stop at answering a single question. They carry out long tasks that span many steps, like fixing code or conducting research. In work like this, countless choices come up along the way: "Which hypothesis should I test first?", "Which implementation should I build this feature on top of?" The catch is that each of these intermediate choices can make or break the final outcome.
Think of it like a climber ascending a mountain. There are many forking paths to the summit; some lead to a cliff, others to the top. A climber's skill isn't just about stamina, it's heavily shaped by the "taste" to recognize which fork is the good one.
This is exactly where existing benchmarks fall short. Evaluation tools up to now have only measured whether an agent runs a task from start to finish and ultimately succeeds (end-to-end success). In other words, they only check whether you reached the summit. But as the abstract points out, no benchmark has isolated and measured an agent's taste itself, its judgment at the forks. If you look only at the final success, you can't tell whether that success came from good judgment or from luck.
The authors' solution is a benchmark called Taste-Bench. The key move is to extract "decision forks" from the trajectories (the footprints of the work process) that real agents leave behind while performing engineering and research tasks.
A decision fork is a point in a trajectory where the path could go in multiple directions and one of them leads to a better outcome. The model being evaluated must pick the better direction at this fork, without seeing what happens after the choice is made. It's like an exam question that asks "what would you do in this situation?" without revealing the consequences of the right answer in advance. To eliminate the bias where the answer shifts depending on the order in which the options are shown, the same problem is posed twice, in forward and reverse order, and only counts as correct if the model gets both right. That's why random guessing yields an expected accuracy of 25%, not 50%.
The clever part here is that these fork problems are mined automatically, with no human annotation. There are two methods.
Because forks are mined automatically from real work logs this way, you can generate a large volume of taste problems without anyone manually labeling "this is an important decision point."
The abstract presents three main findings.
| Finding | Detail |
|---|---|
| Low accuracy | When frontier models were evaluated, even the best got only 59.7% of problems right (must be correct in both orderings to count; random guessing expects 25%) |
| Late-revealing evidence is harder | The more a fork's decisive evidence appears later in the trajectory, the harder it was for every model |
| More reasoning budget doesn't help accuracy | Giving a larger reasoning budget (more time and tokens to think) didn't improve accuracy |
The first finding carries weight. As noted above, random guessing yields an expected accuracy of 25%, so 59.7% is clearly better than chance. But flip it around and it also means models still get roughly 40% of forks wrong. Even powerful models that readily succeed at the final task still have a long way to go on the pure judgment of "picking the right direction at the fork." It's a numeric demonstration that capability and taste can be separate things.
The second and third findings are telling when read together. When the evidence that determines which choice is right only surfaces much later, the model is especially vulnerable. And this problem isn't solved by making the model "think longer." That suggests a lack of taste isn't something you can simply paper over with more compute; it's a different kind of ability.
Finally, the most hopeful result: "taste can be trained." The authors distilled (a training technique for transferring knowledge) the judgments of a teacher model, which sees the outcome (it judges while knowing what happens after the fork), into a student model. As a result, the student model made better decisions on tasks it had never seen before, and even improved its end-to-end success on a held-out set of SWE-bench Pro tasks. In effect, they showed that teaching taste raises real-world performance.
(The abstract doesn't specify the student model's concrete improvement figures or the dataset size, so here we describe it only qualitatively.)
To be honest, there's a lot the abstract alone doesn't confirm.
ONTLE is a service that dispatches AI project managers (PMs). An AI PM inherently performs long-breath tasks, repeatedly making fork choices at every turn like "which task should I handle first?" or "should I keep pushing this approach or change direction?" This paper's framing hits the very core of what ONTLE does.
First, the lesson that evaluating an agent by "final success rate" alone is insufficient. Even if an AI PM ended up wrapping a project up, whether its judgment along the way was good is a separate matter. It suggests that when ONTLE manages agent quality, we need a lens that looks separately at the quality of intermediate decisions, not just the final deliverable.
Second, the result that "taste can be trained" is encouraging in practical terms. The approach of extracting outcome-aware judgments from trajectories and teaching them to the next agent could be the blueprint for a pipeline in which ONTLE treats the success and failure trajectories accumulated in real operation as assets, making its agents progressively smarter.
Third, the finding that "more reasoning budget doesn't improve taste" matters from a cost standpoint. Blindly giving an agent with poor judgment more thinking time may only raise costs without being a solution. It points toward directions where showing more evidence via RAG (retrieval-augmented generation), or training good judgment, may be more effective.
That said, since this paper's results come from engineering and research trajectories, we don't claim they apply as-is to ONTLE's actual work domains. The honest stance is to treat this research as a framework for thinking about "what to measure and how to improve it."
In this paper, taste refers to the ability to pick the direction that leads to a better outcome at the various forks encountered while performing a long-horizon task. The focus is on the "quality of intermediate choices" rather than the final result.
No. Forks are mined automatically from the work trajectories that real agents leave behind, either by comparing multiple parallel attempts at the same task or by finding detours within a single trajectory. The distinctive feature is that it's built without human annotation.
According to the abstract, even the best model got only 59.7% right (and since it must be correct in both orderings, random guessing sits around 25%), and increasing the reasoning budget didn't improve accuracy. That said, improvement was possible by distilling the judgments of an outcome-aware teacher into a student model.
It suggests we need to evaluate the quality of intermediate decisions separately, not just the final success rate, and it raises the possibility of improving an agent's judgment by treating accumulated success and failure trajectories as training assets.