One big reason data agents — AI that digs through databases and documents to answer questions in a human's place — so often lose their way isn't a lack of reasoning power in the model. It's that the data they actually need to work with sits scattered outside the agent, and its structure and contents can't be known in advance. The paper calls this mismatch the agent–data gap, the gulf between the agent and the data.
EvoOntology is an attempt to close that gap. Instead of a static data dictionary written out by hand, it proposes an ontology layer — a kind of guide map that records what the data means — that the agent asks about and repairs for itself while it works. To put the conclusion up front: it clearly improves on a baseline running the same model with no ontology. That said, the abstract's phrasing that it "consistently outperforms the semantic-layer baseline" doesn't quite match the experiment tables, and we'll look at that part together with the numbers later.
Real-world data lives outside the agent, scattered around. Its forms vary too — from relational DBs to semi-structured documents to unstructured files. The agent can barely reach this data at all, and only through generic tools like a SQL interface or a file reader. The trouble is that neither the structure nor the contents of this data is known ahead of time. So the agent throws repeated probe queries (trial queries that poke at a slice of the data), guesses where the concept it wants might live, and digs through irrelevant material in a kind of blind search.
There are broadly two existing approaches to closing this agent–data gap, in which the agent gropes around blindly.
One is raw querying, where the agent reads the schema (the blueprint of the data's structure) directly and fires off exploratory queries. It works well on small, simple data, but on broad, heterogeneous data it easily gets trapped in inefficient exploration.
The other is the semantic layer, where domain-meaning information — schemas, entities, metrics — is built in advance and handed to the agent in one lump. But when the data is large, fitting the entire description into the prompt is impossible given length limits. On top of that, it is usually maintained by hand, so it is expensive to build and hard to keep up to date.
The paper argues that what's needed goes beyond a "static description": an interactive ontology layer that the agent reaches into flexibly through tools.
Imagine a new analyst shows up for work, and the materials are scattered across every department — in filing cabinets, spreadsheets, and databases. A static semantic layer is like a thick in-house handbook someone wrote long ago. It's too long to read cover to cover, it's already out of date, and it has no idea how this particular analyst actually works.
EvoOntology opts for a "reference desk that takes questions" instead of the handbook. When you need something, you ask, and it pulls out only the relevant cards; it builds those cards by actually opening the drawers to check them; and it watches where the analysts get stuck and revises and grows them.
So how is this "reference desk" actually designed? The paper implements it in three layers.
The crux is wrapping these three layers in a single MCP server (the standard way a model calls and uses external tools). Instead of passively being fed metadata, the agent is made to actively ask for it and pull it out.
Picture an agent handed the job: "pull last month's revenue from the customer data."
The first stage is "drawing the initial map." With no answer key, the builder agent looks only at the training-set task logs and the raw data, and shortlists frequently appearing concepts — things like "revenue," "customer," "period" — as candidates. It then fires off its own probe queries and goes through a kind of due-diligence check, confirming one by one whether "revenue" really lives in this table and this column, and whether the values and types make sense. Only candidates backed by the probes survive as cards, and the actual records that grounded them are attached as evidence. Only what's been confirmed, not what's guessed, goes onto the map.
The second stage is "redrawing the map." The evolution agent revises the map using the records of how agents actually worked (the trajectories) as evidence. Successful records reveal which cards and which paths worked; stuck records expose what was missing or confusingly written. For example, if the "revenue" card is missing the rule for handling refunds and the agent gets it wrong again and again, that pattern is extracted and the card is fixed.
There's one important design principle here. A revision touches only one layer at a time. It decides which of content, tool, or schema to fix, and changes just that one place. This is a safeguard to trace the cause of a change clearly and to prevent a fix in one spot from blowing up as a side effect somewhere else.
Whether the fix actually made things better is judged by measurement alone. For each backbone (the LLM that serves as the agent's brain), the revised version is pitted against the pre-revision state on the same validation set under the same conditions, and it's adopted only when it improves by more than a set margin. If it drops, it's discarded, and even the discarded attempt is kept in the log so the same wasted effort isn't repeated. Since each backbone grows independently, an adopted revision reflects that backbone's habits.
Before getting into the main point, let's unpack two terms. Every experiment in this paper is run under the same reasoning scaffold (ReAct, where the model alternates between "think → use a tool → look at the result" and thinking again) and the same compute budget (decoding and interaction budget). In other words, conditions were tightly controlled across the things being compared. Only then can you say a score gap is "the difference the ontology made."
Evaluation was done on three benchmarks. Here's just the gist of each.
Here's the key point. The abstract states that EvoOntology "consistently and substantially" outperforms a strong baseline and the semantic-layer baseline (Baseline + SL). But look at the tables and that isn't always so. On InsightBench, for two models EvoOntology and Baseline + SL (the semantic layer) were equal across every metric.
| InsightBench (Overall) | EvoOntology | Baseline + SL |
|---|---|---|
| GPT-5.5 | 51.0 | 51.0 |
| Claude-Opus-4.8 | 53.2 | 53.2 |
Under these conditions it's a tie, not a blowout. So pulling the results together, it's more accurate to frame it this way. It is clearly better than the same backbone with no ontology attached, but against the condition where the builder agent's generated semantic layer is injected as a static prompt (Baseline + SL), it's a "conditional edge" that sometimes comes out even. One caveat to add: because the scoring systems differ by benchmark (accuracy, insight score, EX/VES), you can't read the numbers from different experiments as one combined figure.
A note on the experimental setup. Here, Baseline + SL is not a pre-existing, human-built semantic layer but the condition where the builder agent's generated semantic layer is injected as a static prompt. So reading this tie as "superiority/inferiority over human-built semantic layers in general, which are expensive to make" would distort the scope of the comparison. BIRD is run in the official Oracle Knowledge setting (the official mode where the background knowledge needed for the question is supplied alongside it), using execution accuracy (EX) as the primary metric and the valid-efficiency score (VES) as the secondary one. The abstract says there are four backbones, but the experimental setup lists six (GPT-5.5, GPT-5.6-sol, Claude-Sonnet-5, Claude-Opus-4.8, DeepSeek-V4-Flash, Qwen3.5-Flash). The figures in the table above cite only the tie values that can be confirmed directly from the paper's InsightBench table.
The very fact that a tie shows up raises a question. Where exactly does this evolution machinery actually earn its keep? By the paper's own logic the gain should grow the broader and more heterogeneous the data gets, but the tables alone don't make that boundary sharp.
On top of that, initialization, evolution, and validation are none of them free — they cost extra compute (the appendix treats the cost of the ontology layer separately). Because each backbone grows on its own, a revision is tailored to that backbone. The paper tried transferring an ontology grown on one backbone to another (Figure 5b); the transfer itself works, but performance dropped somewhat. That is, it means growing per backbone is advantageous — not that "if you swap models you must necessarily grow it from scratch" has been proven.
Also, the builder needs training-set task logs, and the evolution agent needs past trajectories. For brand-new data with no execution history at all, a cold-start (no accumulated records, so starting is daunting) problem remains. The evaluation also spans specific scenarios (e.g., DDR's 10-K), so whether it generalizes is an open question.
This is a problem we at ONTLE run into on the ground every day. We handle data in isolation per customer project, and we bolt on our customers' external systems — their ERP, their shopping malls, their in-house spreadsheets — as we go, case by case. Every time we take on a new customer, the same thing repeats: a human has to ask all over again, "for this company, which table and which column is 'revenue' in, and how do refunds get subtracted out?" The agent–data gap this paper talks about is exactly that scene.
Two things draw us in. One is grounded initialization — instead of a human hand-writing a data dictionary for each customer, you poke at the real data to confirm things and build it together with that evidence. The other is trajectory-based evolution — the picture where understanding of that customer's data improves on its own, out of the work logs, the more it's used. The gate in particular is worth watching: "adopt a revision only when measurement shows it's better, and keep even what you discarded on the record." It stops the agent from quietly rewriting its own knowledge where no one can see.
That said, you have to weigh cost against benefit before bringing it in as-is. For a customer whose data is small and simple, a single well-organized data dictionary may be enough. The evolution machinery looks like it earns its keep when the stack is broad and heterogeneous — but as we saw, that boundary isn't sharp yet. For a product that swaps among several models depending on the task, the trade-off between the performance loss of moving an ontology between backbones and the benefit of growing it per backbone also has to be reckoned with in practice. Rather than rushing to a conclusion, the realistic move is to first try just grounded initialization in one or two places where the data is broad.