When we talk about an AI agent "researching on its own," we usually picture it typing a question into a search box and summarizing the top few links. The challenge this paper poses goes far beyond that. Pull a clue from a video, roam the live web — the real sites that are changing even as you read this — opening different sites, documents, and datasets yourself to stitch evidence together, and arrive at a single answer that holds up as fact.
Here, Deep Search refers to the investigative process in which an AI searches on its own, opens sources, gathers evidence, and works its way to an answer. The researchers assembled a benchmark of 200 such deep-search tasks, OpenDS-World, and gave the same tasks to 12 state-of-the-art models and to humans. The tools in hand, though, differed — the models got research tools like search and a browser, the humans got nothing but an ordinary browser. The best model (GPT-6 Astra) scored 39.0%; humans averaged 21.1% across 1,000 independent attempts. Both fall well short of half.
The paper's example is vivid. Picture a page on the statistics site Our World in Data. On a single screen, explanatory text sits alongside a chart image, and below them the same figures are attached as data files in several formats.
Now suppose you have to pull out "a single value for a particular region in a particular year." That number isn't sitting right there on the screen. You have to pick the region, set the time range, download the table that this filtering produces, and only then fish out the value you need. For a person, it's a few clicks and a quick scan.
But for an agent, every step is a judgment call. It's a maze where you have to decide for yourself "what do I open now, and how." Each individual click looks familiar, but which sequence of those familiar actions — and how many of them — gets you to the answer is an entirely different problem. And most of the real web looks like this. The same fact is scattered as a graph image in one place, podcast audio in another, and in yet another a data file that only opens once you download it.
It's not that earlier deep-search evaluations ignored these difficulties. Information embedded in images and video, navigation that requires manipulating a dynamic interface, interaction that spans multiple websites — each has been handled in its own right. Evaluation methods have steadily advanced through open-web search, multi-turn search, and reasoning that weaves together several sources. What was rare, though, was tying all of these difficulties together at once inside a single task. Missing was an evaluation that demanded the ability to "open and extract" material of wildly different media, formats, and access methods, in an integrated way, across a long chain of evidence.
OpenDS-World organizes this complexity along three axes, and builds its tasks so they draw on more than one — though not every task spans all three at once (by the paper's own count, 82.5% of tasks involve multiple modalities, 96.0% multiple formats, and 95.5% multiple access types).
The Our World in Data page above packs all three axes onto one screen. This is where OpenDS-World draws its line: not testing the three axes in isolation, but combining them along a single connected chain of evidence. Spinning a virtual tour, entering conditions, downloading a bundle of data and pulling the number out of it — these manipulations sit at every turn on the road to the answer.
Before the scores, let's be clear about what was measured and how. The models are given only the question, delivered as video, audio, or text — neither the answer nor the path of evidence that leads to it. In return, every model gets the same tools (web search, browser manipulation, processing to extract evidence from retrieved material) and the same rules. Humans get only an ordinary browser, no LLM tools. An answer is scored against whether it matches "the single fact it resolves to."
Under these conditions, the first thing that stands out is how low the absolute scores are. The top-performing model, GPT-6 Astra, managed just 39.0% across all 200 tasks. Humans were lower still. Across 1,000 attempts — 50 people, five independently on each task — the average was 21.1%.
Don't read these two numbers as "AI beat humans." They are separate measurements under different conditions to begin with. The 39.0% is one model's accuracy over 200 problems; the 21.1% is the average over 1,000 human attempts, with people going in bare-handed, no LLM tools. That the model came out higher means the advantages of automated exploration and high-volume reading are clear in this class of tasks — not that it is better at research in general. And the fact that first place is 39% in absolute terms is a signal that "deep research on the real web" still has a long way to go.
Break the human results down by problem and the difficulty is uneven. A little over half the tasks were solved by at least one of the five people, but on nearly the other half, all five came up empty. The time invested is laid out in the table below too.
| Metric | Value | How to read it |
|---|---|---|
| At least 1 of 5 succeeded | 115 (57.5%) | More than half the tasks are at the "someone can crack it" level |
| All 5 failed | 85 (42.5%) | That many tasks went unsolved by anyone |
| Average time per attempt | ~38.8 min | This is how long one problem took |
Only 1 task (0.5%) was solved by all five, so there were essentially no "easy" problems. Even majority success (3 or more) came to just 29 tasks (14.5%), so "reliably solvable" tasks are under 15%. In terms of labor, each person (20 problems) spent an average of 12.92 hours and a median of 9.44 hours, totaling about 646 hours. Subjective difficulty (0 easy to 5 hard) averaged 3.884 with a median of 4; scores of 4-5 made up 65.5% (5 at 43.0%, 4 at 22.5%), and not a single task was rated 0 (easy).
There's one observation worth more attention than the scores themselves: calling more tools and spending more runtime did not necessarily raise accuracy. The most common intuition when automating research — "let it search more and it'll find more" — is exactly what the data sidesteps.
Why would that be? From here on is interpretation aimed at making sense of the results (what the paper actually measured stops at the fact that calls and time don't scale with accuracy). Several possibilities overlap. One is that research runs like a chain. Pick the wrong source at an early step and, no matter how much you dig afterward, you only pile up "grounds propping up a wrong answer." A rising search count can itself be accumulated flailing. But other explanations are possible too. The problem may have been hard to begin with, which drove the call count up; or the agent couldn't reach the material and kept repeating the same attempt, inflating the calls. The bare fact that calls are high can't separate these three. To tell which it is, you'd have to log, step by step, "at which stage, what got stuck" — a homework assignment we'll meet again in "Limits and Open Questions" below.
What's clear is that the point isn't digging a lot but digging well. The crux lies in judgments like which source to open, whether to verify mid-stream before moving on, and when to stop. The authors likewise read the cause as limits in source selection, tool orchestration, search-strategy adjustment, and long-horizon management (the ability to carry a multi-stage investigation all the way to the end). The paper also includes analysis of aspects like cost, stopping point, failure recovery, and budget.
To wring out scores like these, the tasks themselves have to be built in a special way. The heart of the OpenDS-World design is that tasks are built backwards. Normally you write the question first and then look for the answer; here, you first weave a chain of evidence, then invert that chain into a question. Put that way it sounds abstract, so let's follow along with a made-up example for the sake of explanation. Say a certain building appears in a video.
Step 1 — Fixing the starting point (Real-World Anchor). Every task starts from a single publicly available video. An annotator analyzes the visuals and audio together and singles out one target "that can only be identified from this video." In our example, that's the building on screen. The key condition: the building cannot be identified by reading the text on the video's page alone — you must watch the video and listen to the audio to pin it down. That's what keeps the task from being solved by plain text search. The building fixed this way must correspond uniquely to one Wikidata entry. Wikidata is a public database that assigns a unique number to each person, place, and thing in the world; nailing it down to one entry here keeps "that building" unmistakably a single target and prevents the answer from turning ambiguous later.
Step 2 — Extending the chain (Open-Web Evidence Expansion). Starting from that building, the annotator searches for and links the next target. For instance, confirm the building's architect (a person), then move on to another of that architect's works (a target). With each hop, the material is deliberately chosen so the modality, format, and access type vary. And each link is cross-checked against several independent sources rather than resting on one. This continues until it reaches a stable endpoint.
Step 3 — Inverting into a question (Question Inversion). With the final target set as the answer, the entire chain built so far is turned into interlocking conditions that form the question. The building, architect, and work in the middle aren't named outright but are veiled as "indirect references" that must be inferred from the preceding evidence. No single clue points straight at the answer, but as the conditions pile up the candidates narrow until they converge on a single answer. Searching the stated words verbatim won't solve it; you can only solve it by reconstructing the hidden links between the intermediate targets and tracing the chain back.
Two layers of review are added on top. Ten independent reviewers retrace the chain through the preserved source links, checking both whether the evidence holds (factual accuracy, cross-checking across multiple sources, grounds for the links between nodes) and whether the task holds (clarity of the question, consistency of the conditions, uniqueness of the answer). Cases with factual errors, insufficient evidence, or more than one answer are discarded. Only the 200 that passed this filter remained in the final benchmark.
The evaluation runs 12 models with the same tools and the same rules. Because tool access was unified, you can isolate the fact that differences in scores come from "how you researched," not "whether you had good tools." So what this score measures isn't "skill at a single search." It's the ability to orchestrate multiple media, formats, and manipulations at every step and carry a long chain all the way to the end. In fact, 82.5% of the individual tasks mix two or more modalities. What the material covers, and how much, is laid out in the table below.
| Item | Value | How to read it |
|---|---|---|
| Annotated material items | 787, averaging 3.94 per task | Solving one problem means passing through close to four places on average |
| Coverage | 4 modalities, 42 formats, 7 access types | It spreads wide — not just media, but file structures and manipulation methods too |
| Modality appearance rate | Text 99.5%, image 44.5%, video 35.5%, audio 16.0% | Text is nearly always in the mix, but the evidence doesn't live in text alone |
The limit the authors clearly acknowledge is that current agents keep wobbling on tool orchestration and long-horizon management. The fact that strategies vary widely from model to model and that those strategies don't translate into accuracy means no stable method has yet been established.
There are limits that come from the design, too. One is reproducibility. Because the tasks assume a live web, pages may change or disappear over time. The researchers state that they archive the starting-point video and distribute it along with the task input, but fixing every external source in the middle of the chain is inherently difficult. Another is the nature of the answers. A design carved down so the answer resolves to a single unique fact makes scoring clean, but it leaves outside its scope the kind of question common in real research — where there are multiple right answers or judgment is required. The shared format of starting from a video itself also introduces a particular bias into the inputs.
And whether 39% or 21%, this score doesn't tell you "at which stage, and why, things collapse." We said earlier that call count alone can't separate flailing from difficulty from access failure — that very distinction is at stake here as well. Whether the chain breaks at a particular node, whether it gets stuck switching modalities or stuck on an interactive manipulation, is hard to tell from these results alone. The clues only emerge when you look alongside analyses by domain, by attribute, and by video length.
How would this result apply to ONTLE's research feature? From here on is our own judgment, not the paper's conclusion.
Suppose ONTLE's AI PM gets this instruction: "Pull together the pricing plans of three competing services, including the quarterly trend." In reality, this information is scattered — in one place inside a table in a product-brochure PDF, in another behind a pricing calculator where numbers only appear once you enter region and seat count, and in yet another in the on-screen captions of a YouTube demo video. It's a task that layers together exactly what we saw earlier — "multiple media plus access that requires manipulation" — one that summarizing the top search links can't reach.
On top of that, "quarterly trend" demands past values. Not the price on the screen right now, but how much it was under which region and seat conditions in each quarter, and where you saw that number — only when you record all of that does it become a "trend." Let even one condition slip and you end up lining up numbers measured against different baselines in a single row.
So the design we've chosen is this (not the way it's currently implemented, but the direction we set after reading this paper). Rather than handing a long investigation to one smart agent that finishes it in a single shot, build a checkpoint-style pipeline that breaks it up and verifies at each node. When the PM agent fetches the quarterly figures, it attaches to each value the query conditions (region, seat count, point in time) and the source link, and any value that hasn't been cross-checked is flagged as "unverified" and escalated to the human PM.
What matters is how you handle the ones you couldn't fill in. If you ultimately couldn't find the past price for a given quarter, leave it not as a blank but as "acquisition failed"; if you found a source but the number's seat conditions don't match, flag it separately as "condition mismatch." That way the human can see at a glance what still needs checking.
Do this, and the human PM can look only at the flagged parts instead of combing the whole thing from scratch. We expect review time to drop substantially, but that's still a hypothesis we have to verify, not a measured effect. At the very least, the direction is clear: rather than automation erasing the human's work, it leaves the points where the human must judge sharply marked out.
Here are the things to actually check when auditing research automation.
Format refers to the file and data structure the material lives in — the range is wide, from web pages, documents, and image or audio files to data files you have to download and open in a dedicated tool. Access type is a classification of the manipulation needed to extract the evidence: page navigation, entering conditions, specifying a period, manipulating a view, structured queries, resource downloads, and so on. The key point is that even the same "text" varies in what structure it's in and how you have to access it, and this benchmark deliberately laid that range out wide.