There's a new way for language models to gauge how much they should trust their own answers. The idea is simple: pull up the track record the model has built over time—a kind of report card—and use it to score "how likely is this answer to be correct." The authors call it XConf (eXperiential Confidence). To cut to the results: across nine benchmarks, it estimates confidence better than existing approaches on most of them, and at a fraction of the cost.
Whenever you put a language model into a real product, the same question comes up: should this answer just ship, get handed to a human, or be retried? The basis for that call is confidence estimation. When a model says "this answer is 90% right," it really needs to be right nine times out of ten before you can trust that number enough to automate around it. The property of a stated probability actually matching the real success rate is called calibration.
Why calibration matters becomes obvious the moment things go wrong. If an answer the model claimed it was "90% confident" in turns out to be right only half the time, any automation built on that number either misfires wholesale or bleeds cost. Confidence isn't just a reference figure—it's the safety mechanism for automation.
The catch is that existing confidence-estimation techniques all share the same weakness: they look only at the single reasoning pass happening right now. They come in three flavors. You can ask the model directly, "how confident are you in this?"; you can score the probabilities of the tokens it generated; or you can make it solve the same question several times and see how much the answers agree. That third one is the widely used self-consistency approach—and solving a question ten more times means ten times the generation cost.
What all three are missing is the model's own history. Picture a student in an exam trying to judge "did I get this one right?" by staring only at that single problem. The decisive piece of information—"I've said I was confident on problems like this and then been wrong a lot"—is completely absent. People, once they've built up experience, know "when I'm this confident I'm usually right" or "this is exactly the kind of situation where I get careless." XConf fills in that blank.
XConf hands the model a personal report card and, when it meets a new problem, has it open that card and look. Let's see how it works first, then why it's practical.
It starts with how experience gets stored. The model leaves behind each problem it has solved as an episode. Just like a student keeping a mistake log, it writes down what the problem was (the task), what it was thinking at the time (the reflection), how confident it was (the confidence), how it turned out (whether it was correct), and finally what it learned once the answer was graded (the lesson)—all as a single "experience." Lines like "this type, I was 80% confident, got it wrong, cause: missed one condition" pile up into a report card.
When a new task comes in, it works in two stages. First, in the Recall stage, it pulls up past episodes that were a similar type to the current task and tackled with a similar level of confidence, and reads how many of them actually turned out correct (the historical success rate). It checks against data: "when I said I was this confident before, how often was I actually right?" Then, in the Reflect stage, the model looks at that record, names the failure patterns it keeps repeating, and restates its confidence in light of that history. What comes out isn't naive confidence in the answer it just produced, but confidence corrected by its own track record—something like "I tend to overrate myself in situations like this, so I'll state it lower."
This structure is a twist on a familiar paradigm. If retrieval-augmented generation (RAG, the technique of retrieving external knowledge and pulling it into the answer) is an external memory store for knowledge, XConf is an external memory store for judgment. All that's changed is what gets pulled in—from outside knowledge to one's own history—while the skeleton of retrieve-and-use stays the same. And that small difference is telling. It's a sign that AI is moving beyond simply getting smarter and into a metacognitive stage, where it looks back on its own past to check itself.
Three strengths follow from how it works. First, it's format-general. Whether the answer is a number, code, or free-form prose, it applies as-is. Second, you don't need to crack the model open. There's no accessing token probabilities (logits) or retraining weights, so it drops straight onto closed, commercial models you can only reach through an API. Third, it amounts to generating an answer once and then making one short reflection call—a different weight class from self-consistency, which re-solves the same question ten times.
The evaluation spanned nine benchmarks across reasoning, coding, multimodal QA (answering questions over both images and text), and interactive agents (AI that solves tasks over multiple steps while using tools), plus four models drawn from three families. The results break down along three axes.
First, discrimination—how well confidence separates right answers from wrong ones, measured by AUROC. On the non-agent tasks, where the model answers in one shot, XConf was pitted against 10-sample self-consistency across 6 benchmarks and 4 models, for 24 combinations. The outcome was 21 wins, 2 ties, 1 loss—ahead of or on par with self-consistency on 23 of 24. For the 3 agent-task benchmarks, instead of re-running 10 times, it was compared against separate baselines.
Next, calibration error. ECE (Expected Calibration Error), which measures the gap between stated probability and actual success rate, was much lower. In other words, "90% confident" landed much closer to an actual 90%.
Cost is the thread that ties the beginning of this piece to its end. XConf delivered the above at one-tenth the number of answer generations compared to self-consistency. That said, this "one-tenth" counts only answer generations. Recalling the past requires embeddings (turning text into numeric vectors to find similar ones) and retrieval, and that cost is spelled out separately in Appendix L—so the total API bill doesn't drop to exactly one-tenth. Even so, it's far more efficient overall than re-solving the same question ten times.
The result that lands most directly in practice comes from selective prediction. One caveat first: the discrimination and calibration gains above are about estimating confidence well, not about the answers themselves becoming more accurate. Selective prediction shows what that well-estimated confidence is good for. When XConf abstained from answering on the bottom 10% of episodes by confidence, the success rate of the answers actually delivered rose by up to 8.7 points on agent tasks. Just filtering out the ones it wasn't sure about visibly improved delivery quality. It's a number that plugs straight into the operational call of what to ship, what to escalate, and what to retry—and the effect was especially pronounced on multi-step, tool-using agent tasks.
Better-estimated confidence at lower cost—that's the crux of why experiential confidence estimation is a leading candidate to become the next general-purpose approach to confidence estimation.
There are a few structural limits worth examining before you put this into a real product. What follows isn't quantified in the abstract; these are questions that follow from the method's design.
ONTLE is a service that dispatches an AI project manager to each client project. If an AI PM is going to judge and hand off work in a person's place, the most dangerous moment is when it's wrong but pushes ahead with confidence. So what happens if ONTLE's AI PM is equipped with XConf?
Picture the moment a client requests an integration with an external system and the AI PM is about to answer, "this'll take half a day." Before shipping that answer, the model opens its own report card: "on integrations that looked this simple, how often did I actually hit the schedule?" If it has made the same confident boast before and been wrong a lot, it lowers its own confidence and routes the item to a human owner instead of handling it automatically (escalation—passing an ambiguous case up to a higher-level owner). The flow of look up the past record, revise confidence, hand off to a human maps directly onto an abstain-and-escalate design. The ambiguous ones go to a person; the sure ones get automated.
As it happens, a product like ONTLE already accumulates project history and work results. XConf aligns with the idea of repurposing that history as an "experience store" for confidence correction—and the fact that it needs no logit access and no retraining, just one short reflection call, fits well with a product environment that bolts commercial models on via API.
That said, the paper's benchmark results are not a guarantee for this workflow. Applying a bottom-10% abstention to real-world work automation makes the limitations above the very things that need validating. Is there enough of a report card built up (cold start)? Is the task distribution stable (distribution shift)? Does it correctly judge "similar tasks" (the similarity criterion)? You have to nail those down first before you can trust the abstain call and hand automation to it. Even so, the direction—grading confidence from accumulated experience rather than a single reasoning pass—is a clear reference point for any product aiming at automation you can actually trust.
No. It works without weight updates or logit access (per §5.1 of the paper), so it can be applied as-is, with no extra training, even to closed models you reach only through an API.
It cuts the number of answer generations to one-tenth. That said, it adds the embedding and retrieval cost of recalling the past (Appendix L), so the total API bill doesn't come out to exactly one-tenth. Even so, it's more efficient than re-solving the same question ten times.