A single line lands in the chat box.
"That timeout — do we have anything written up on what caused it?"
A few seconds later the AI PM answers. It lays out why payments kept stalling, when the failures clustered, and how the team decided to respond — point by point — and at the very end it names the documents it drew on.
Which is odd. This AI never joined your company and never looked at your payment module. So how does it answer as if it knows "your project"? Not because it's clever. It's because every time a question comes in, a process runs behind the scenes that fetches the relevant documents and folds them into the answer. That process has a name: RAG (retrieval-augmented generation). Instead of leaning on knowledge memorized in its own head, the AI looks up outside documents on the spot and answers with those in hand.
Picture a sharp consultant on their first day. Strategy, finance, marketing theory — they've got it down cold. But ask them "How many unused vacation days roll over here?" and they draw a blank. Not because they aren't smart. They just haven't read this company's policy handbook yet.
A good hire doesn't make something up here. They say "Hold on, let me check the handbook," pull the relevant clause from the HR folder, and answer while pointing at the page. That is exactly the three moves RAG makes. A question comes in, it finds the relevant documents, lays them on the desk alongside the question and hands the whole thing to the model, and the model reads what's laid out and writes the answer. Behind the scenes these three steps are called retrieval, augmentation, and generation. You don't need the jargon. Find, lay out, write. That's the whole thing.
A large language model (LLM — an AI trained on enormous amounts of text) absorbs whatever had piled up on the internet by the time of training. So it's good at general knowledge, coding syntax, writing tips — the equivalent of the consultant's head full of theory. Call this the model's in-head knowledge.
The trouble is the spec your team wrote yesterday, the rule set in last week's meeting, the terms only your company uses. None of that was in the training data to begin with. It can't know what was never there. Worse, it fakes it even when it doesn't know — confidently serving up an answer that sounds right but is wrong. This is called hallucination.
So you have to keep knowledge in two places. One is the in-head knowledge just mentioned. Once it sets, picking out a single piece of it to fix is hard. The other sits outside the model, on the desk: the documents. Swap a document out of the storeroom and the content changes instantly. Your project's documents live here.
The way you solve the "doesn't know your project" problem forks right here, too. One road is to retrain the model on your data and rewrite the in-head knowledge itself (fine-tuning). It costs money and time, and it's a poor fit for a project whose documents change several times a day. You can't re-bake the model every time today's schedule shifts. The other road is RAG. You leave the model alone and just swap the documents on the desk. Stock the storeroom, and the model pulls from it whenever a question comes in.
To put it simply: the model handles the "how to think," the documents handle the "facts that are ours." Just as the new hire uses theory and the handbook together.
For the new hire to find the handbook, the handbook has to be filed in a folder. RAG likewise gets its findable material ready before any question arrives.
First it breaks documents into pieces. A single spec runs dozens of pages. Whole, it's hard to search and hard to feed to the model. So you split it into small morsels — chunks (a chunk being one slice cut from a document). That way you can pull only the slice that touches the question, and it fits within the amount the model can read in one go.
The catch is where you cut. If one page of the handbook gets torn at "For the three reasons above, the schedule will be—" and the next piece starts with "—pushed two weeks," neither piece makes sense on its own.
How much does the cut point alone change the answer? The weak approach slices mechanically at, say, every 500 characters. A sentence gets severed down the middle and the morsel loses its context. The strong approach cuts at paragraph and heading boundaries and overlaps a sentence or two between neighboring morsels. The overlap stitches the broken context back together, so even one morsel read alone holds up.
Once cut, you file it for easy retrieval. You build a list of which word sits in which piece (an index), and you also store a version of each piece with its meaning turned into a row of numbers (a vector). Why keep two copies becomes clear in the retrieval step, coming up.
Setup done. Back to that opening scene. The user, who's been talking about the payment module, asks:
"That timeout — do we have anything written up on what caused it?"
A timeout is when you send a request to an outside system and, if no reply comes by a set deadline, you treat it as a failure. Meaning payments kept failing that way. This one line travels through four steps and comes back as a sourced answer.
"That timeout" alone doesn't tell you what to look for. What "that" points to lives in the conversation that just happened. So when building the search query, you look at two things together: the question that just came in, and the conversation right before it.
Here the previous topic was the payment module, so you expand "that timeout" into "payment API timeout" and build the query from that. (An API is the set of rules programs use to pass data to each other.) It's the same as the new hire parsing "when's that due again?" against what you were just discussing.
What happens if you miss what "that" refers to? The weak approach searches "that timeout" verbatim, with no context, and dredges up the wrong "that." The strong approach adds the key name from the recent conversation (payment module) to the query and narrows in precisely. But lean too hard on past conversation and the search gets dragged toward a topic that's already closed, so you stop at lightly folding in the recent context.
With the query set, you pull the relevant pieces from the document storeroom. There are two routes for pulling, and both map cleanly onto how the new hire searches the handbook.
One is searching by word. Like the new hire digging through folders with an exact phrase such as "Article 14" or "payment API timeout." In technical terms this is keyword matching (BM25 is the classic). Fast and precise, but weak when few words overlap. If the document is written only in other terms — "payment request delay," say — then a search for "payment API timeout" shares few words with it and tends to get pushed down the ranking.
The other is searching by meaning. Like the new hire recalling and finding something not by its exact words but by its sense — "the vacation-rollover thing." This is where the rows of numbers (vectors) stored earlier come in. It picks pieces close in meaning to the question, so it links "payment API timeout" and "payment request delay" even though the words differ. In technical terms this is semantic search (dense retrieval). That searching by meaning this way catches relevant documents better than searching by words alone became widely known through the 2020 DPR (Dense Passage Retrieval) research.
So should you just use search-by-meaning? No. When you need an exact error code like "ERR_4092," search-by-word is better. Search-by-meaning can drag in "errors that feel similar," when what you actually need is the one document where that exact code sits. So in practice you run both together. This is called hybrid search. Search-by-word grabs the document containing "ERR_4092" as written, and search-by-meaning catches the differently-worded "payment request delay" document. Combined, even the document each would have missed alone makes it onto the candidate list.
What do small differences in search method lead to? The weak approach searches by meaning only, so even when you ask for "ERR_4092" it floats "similar" errors with different numbers to the top. The strong approach runs word and meaning together, lining up both the document with the exact code and the documents close in meaning as candidates.
With the search done, three pieces came up as candidates from the storeroom (there are really more, but let's look at just three).
Piece 1 — [Payment Module Issue History] "Payment API timeout (ERR_4092): when the external payment provider's response is slow, the request drops straight to failure with no retry. Decided response: add retry logic and raise the response wait time."
Piece 2 — [Payment Meeting Notes] "Payment request delays were especially frequent during peak-traffic hours. Decided to adjust the alert threshold to catch them earlier."
Piece 3 — [Payment Module Release Notes] "Changed the button color and copy on the payment-complete screen; shipped."
Two unfamiliar terms show up in the pieces. Retry logic is a mechanism that doesn't give up on the first failure but sends the request a few more times. The alert threshold is the line for "sound the alarm when the number crosses this." When traffic spikes and delays grow frequent, lowering that line means catching them earlier.
This is where the real design battle starts. You don't just use the candidates as they came up. The search engine scores each piece for relevance — higher the closer it touches the question, lower the more it merely brushes past. You pick and discard by this score.
Piece 1 scores high because "timeout" and "ERR_4092" hit the question head-on. Put it first. Piece 2 describes the same incident in different words. The words don't overlap, but the meaning is close, so the score backs it up. It broadens the cause, so append it after. Piece 3 shares only the word "payment"; it's a stranger to the timeout, so its score is low. Discard it.
This discarding judgment is the crux. In the new hire's terms, it's laying only the two handbook pages that fit the question on the desk and closing the one that doesn't. Spread irrelevant material out alongside and the model reads it too and mixes in off-topic nonsense. So the input actually handed to the model looks roughly like this.
[Instruction] Answer based only on the reference material below. Do not make up anything not in the material; say you don't know. [Reference material] 1) Payment API timeout (ERR_4092): when the external payment provider's response is slow, the request drops straight to failure with no retry. Response = add retry logic + raise response wait time. 2) Payment request delays were especially frequent during peak-traffic hours. Decided to catch them earlier by adjusting the alert threshold. [Question] The payment API timeout — do we have anything written up on the cause?
Finally the model reads the reference material and the question together and writes the answer. The principle is single: ground it in the documents just laid on the desk, not in general knowledge from its head. It's the new hire pointing at the open handbook and saying "if you look here," instead of reciting general theory from memory.
That's why a well-built RAG appends, at the end of the answer, which documents it looked at. In our scene the answer comes back like this.
"The payment API timeout (ERR_4092) is written up as being caused by the request dropping straight to failure with no retry when the external payment provider's response is slow. It was especially frequent during peak-traffic hours. The response on record is to add retry logic, raise the response wait time, and adjust the alert threshold to catch it earlier. (See: Payment Module Issue History, Payment Meeting Notes)"
With sources attached, the user can open the originals before trusting the answer. The opening "do we have anything written up?" has, only now, become a grounded answer.
No matter how well you stock the document storeroom, moments come when the search results are thin or empty altogether. And a well-designed system uses exactly that moment as a chance to grow itself. The fallback (a backup action for when the normal path is blocked) is the device for this.
First, when does the system decide there's "no grounding"? It reuses the relevance scores from before. If even the highest-scoring piece that came up falls below a threshold, it judges there's nothing usable to ground on. It doesn't force-scrape something together.
The biggest danger here is the model filling the blank with general knowledge from its head because the answer isn't in your documents. It's the new hire who, unable to find it in the handbook, invents "we probably do it this way too" from some other company's practice. It sounds plausible for the moment. But if a customer believes it and makes a wrong decision, the responsibility lands back on the company.
A good fallback, instead of making things up, says so honestly: "I couldn't find relevant material. Upload the document and I'll reflect it from next time." That one sentence does two things at once. One, it builds trust by showing it knows how to say it doesn't know. Two, it tells the user where the storeroom is empty. A document uploaded in response to that prompt becomes a search target for the next question. So the fallback isn't a failure message — it's the first turn of a virtuous cycle that thickens the storeroom.
So how does an actual product implement this principle? What we've been calling the "document storeroom" is handled in ONTLE by a feature called the Knowledge ONTLE.
Each project gets its own dedicated Knowledge ONTLE. Upload a document and it's chunked and stored, just as we saw. Ask the AI PM something, and in that instant it searches only within that project's Knowledge ONTLE and folds the results into the answer. Return once more to that payment-timeout scene. When the user types "do we have anything written up on that timeout's cause?", ONTLE pulls the payment issue history and the meeting notes from that project's storeroom, strips out word-only-overlap pieces like the release notes, and answers with sources attached. To the user it's just "an AI PM that knows our project," but behind it those four steps run on every question.
That's why what ONTLE pours its effort into isn't "cramming in more documents" but "picking out the right one." Accuracy doesn't rise on its own as documents pile up. If anything, when irrelevant or stale material gets searched and mixed in, wrong answers can grow (a relevant experiment). Stripping out word-only-overlap material like Piece 3 at the right moment is exactly why this matters.
The fallback is baked into the product, too. When there's no grounding, instead of making things up it admits it couldn't find anything and asks you to beef up the documents. Upload a document — no retraining the model — and your reference material grows: the very upside of RAG, which comes back to documents on the desk.
If you need an AI PM that remembers your project's context, start with ONTLE now. https://ontle.kr
If you have only a few short documents, sure. But once a project's documents pile up, you quickly exceed what the model can read in one go. And pasting everything in every time pulls in material unrelated to the question, which invites wrong answers. RAG selects only the relevant morsels on the spot, so no matter how many documents accumulate, it uses only the material that fits the question.
They serve different purposes. Fine-tuning is for changing "ingrained habits" like tone or answer format. RAG is for supplying "facts" that change often. You can also use them together. That said, trying to cram daily-changing project knowledge in via fine-tuning is miserable to maintain. That kind of knowledge is better kept in the document storeroom.
Adding a lookup does take more effort than pure generation alone. But it doesn't comb the whole storeroom — it narrows candidates using the pre-built index and vectors, then passes only a handful of morsels to the model, so the lag you feel is small. If anything, shoving a long document in whole can be slower, since there's more to read.