You told it your name five minutes ago, and the chatbot asks again. The model isn't being dense. A production large language model (LLM, an AI trained on mountains of human writing to predict the next sentence) holds onto nothing between one request and the next. Every call starts from a blank page. Say "that thing we talked about yesterday," and all the model receives is that one sentence, dropped in with no surroundings.
So why does it feel like a continuous conversation? Because the app reloads the conversation so far as text and hands it to the model on every single request — not necessarily the whole thing, but at least the recent turns, and often a short summary standing in for the older stretch. To you it looks seamless, but behind the server a fresh bundle gets assembled each time: "here's the history that matters, and here's the new question." A chatbot's memory isn't really memory at all. It's closer to a briefing that gets re-read aloud every time.
That narrows the design question down to one thing: where do you keep that history, and when do you pull it back out? The answer is almost boring. You keep it in a database (DB). The genuinely hard part comes next, the moment you bolt on a cache to speed things up.
People usually call this history management a "session." And they wave it off with "sessions are supposed to disappear quickly." Wrong. How long a session lives isn't an intrinsic property of the session. It's decided by where you stored it.
In fact, Django's session backend defaults to the DB. "A restart wipes the conversation" isn't a fact about sessions. It's just the symptom of keeping a session in memory only. Move the storage somewhere durable and the whole class of incident disappears.
The simplest and sturdiest shape is this. Keep the raw conversation in the DB, and on every request, pull the last few turns (a few back-and-forth exchanges) and rebuild the prompt. No cache means nothing to go stale.
def build_prompt(conn, conversation_id, new_message):
# 사용자 발언을 먼저 영구 기록한다 — 응답 생성 중 서버가 죽어도 발언은 유실되지 않는다
append_turn(conn, conversation_id, role="user", content=new_message)
# 원본에서 최근 N턴을 매 요청 조회 — 방금 넣은 발언까지 포함된다
history = fetch_recent_turns(conn, conversation_id, limit=20)
return assemble_prompt(history)
Two principles are hiding in here. One: record the user's message the instant it arrives, not after the response finishes. If the server dies mid-generation, what the user said is still on disk. Two: build the prompt from the source of truth. Whether they pick up on a different device or the server just rotated, you read the original, so it's always current.
For most chatbots, this is enough. You can add a cache later, after you've actually confirmed the reads have gotten slow.
Reads become the bottleneck, and you reach for a cache (temporary storage that keeps frequently used data close by). Here's the obvious-but-wrong idea that shows up at this point: "Let's remember the largest turn number, and if the DB's max is unchanged, just reuse the cache."
Why it doesn't work becomes clear if you follow two people talking at almost the same instant. Say turn numbers climb by one each time.
The max never budged from 11, so the cache concludes "nothing changed." And it keeps serving a history that's missing message 10. The user clearly said A first, yet in the chatbot's reply that statement looks like it evaporated entirely. If message 10 was "please move the deadline up by two weeks," the chatbot acts like a person who never heard the request. Even though the data is sitting right there in the DB.
The nastier part is that this never throws an error. The server returns 200 OK, the logs are clean. The answer is just slightly off. A cache is a shortcut for speed, not a source of truth. If you decide to cut a shortcut, you have to set the rule for closing it when it goes stale, or the accidents come.
From here we work through the actual fix in code. Readers who only want the takeaway from above can skip to the next section.
There's exactly one swap. Instead of "the largest number," use a version counter that increments by one on any change at all. Insert, edit, delete, any change bumps the version. Store that version alongside the history in the cache. On read, compare it against the DB's current version, and if they differ, read again.
This makes the earlier trap disappear. Even if 11 commits first and 10 commits later, the later commit bumps the version one more time. On the next read the versions disagree, so you throw out the cache and re-read the source. It also catches edits and deletes of existing turns, which the max could never see.
def get_history(conn, conversation_id, cache):
cached = cache.get(conversation_id)
with conn.begin():
# 버전과 턴을 '같은 순간의 스냅샷'으로 읽기 위해 격리 수준을 올린다(아래 설명)
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ")
version = conn.execute(
"SELECT version FROM conversations WHERE id = :cid",
{"cid": conversation_id},
).scalar()
if cached and cached["version"] == version:
return cached["turns"] # 버전 일치 → 무거운 턴 조회 자체를 생략
turns = conn.execute(
"SELECT role, content FROM turns "
"WHERE conversation_id = :cid ORDER BY seq",
{"cid": conversation_id},
).all()
cache.set(conversation_id, {"version": version, "turns": turns})
return turns
The read side screens for a cache hit with a cheap version lookup first, and only runs the heavy turn query when they disagree.
It's worth pausing on why the comment says "a snapshot of the same instant." If another write slips in between the moment you read the version and the moment you read the turns, you can end up stamping a mismatched pair into the cache, a version number paired with turns that don't match it. To prevent this, the DB offers a mode where a single transaction (one bundle of operations that all succeed or all get cancelled together) keeps seeing only the data as it looked when the transaction first observed it. In PostgreSQL that level is REPEATABLE READ. Under the default READ COMMITTED, each statement picks a fresh point in time, which opens the gap. The precise semantics are in the PostgreSQL transaction isolation docs.
The version counter tells you whether the cache is stale, but it doesn't erase the separate problem of "the order the numbers were assigned flipping relative to the order they actually committed." In the brief gap between two commits, some read briefly sees a history where 11 exists but 10 doesn't yet. The cache self-corrects on the next read, but if you want to eliminate even that momentary time travel, you lock at the conversation level before the numbers are ever assigned.
def append_turn(conn, conversation_id, role, content):
with conn.begin(): # 커밋 시점에 잠금 해제
# 1) seq 배정 '이전'에 대화 행을 잠근다(FOR UPDATE) → 동시 쓰기가 한 번에 하나씩
conn.execute(
"SELECT id FROM conversations WHERE id = :cid FOR UPDATE",
{"cid": conversation_id},
)
# 2) 잠금 안에서 다음 seq 계산 → 배정 순서와 커밋 순서가 어긋나지 않는다
seq = conn.execute(
"SELECT COALESCE(MAX(seq), 0) + 1 FROM turns WHERE conversation_id = :cid",
{"cid": conversation_id},
).scalar()
conn.execute(
"INSERT INTO turns (conversation_id, seq, role, content) "
"VALUES (:cid, :seq, :role, :content)",
{"cid": conversation_id, "seq": seq, "role": role, "content": content},
)
# 3) 모든 변경에서 버전을 올린다 → 캐시 무효화의 단일 신호
conn.execute(
"UPDATE conversations SET version = version + 1 WHERE id = :cid",
{"cid": conversation_id},
)
FOR UPDATE is a lock that says "no other write touches this row until I'm done with it." When A and B both try to write to the same conversation, only one gets the lock, and the rest wait, unable to so much as compute their own number. The situation where "11 commits before 10" simply can't arise. The lock is per-conversation, so writes to different conversations still flow side by side. The behavioral details are laid out in the PostgreSQL explicit locking docs.
Up to here was the craft of not losing the memory. From here it's the craft of using it well. No matter how precisely you've stacked the history, the quality of the answer turns on how you hand that history to the model.
Once a project runs for a few months, the history crosses hundreds of turns. Cramming all of it into the prompt every time is no answer, which is exactly why real systems rarely re-send the raw full history: they pass the recent turns verbatim and let a summary stand in for the older stretch. Anthropic's own context compaction works this way, replacing older conversation with a summary as the context fills. Liu et al. (2023), in Lost in the Middle, gave one reason placement matters: they reported a U-shaped tendency where, when the key information sits dead center in a long input, the model tends to miss it. It reads the beginning and the end well, but the middle goes fuzzy.
Picture a three-hour meeting. The first agenda item and the final conclusion stay sharp, but the detailed discussion from hour two you barely remember. So the longer the history, the more you need placement, not "stuff it all in." Pin the decisions you've locked in, summarized short, at the very front. Keep the last few turns verbatim at the very end. Compress the old stretches into summaries so the middle passes by quickly. The point is to make the model meet the important information in the spots it reads well.
One more thing. Even within the same project, you should split the conversation window per participant. This isn't convenience, it's privacy. The moment a concern teammate A raised privately bleeds into B's window, trust breaks. In DB design terms, that means the key for finding a history can't be "project ID" alone. It has to be the combination of "project ID + user ID." Only shared information, like the schedule or the deliverables, gets shared separately.
Building ONTLE's AI PM, we stepped right into this trap ourselves. The early version used the "reuse the cache if the largest turn number is unchanged" approach. Day to day it was fine. The problem blew up the instant a customer asked to "move the deadline up by two weeks." The task of writing that message to the DB and the task of the AI PM immediately composing a reply and writing its own message ran at nearly the same time. The max turn number didn't budge, and the cache served the next response a history with the deadline-change turn missing. The AI PM acted as if it had never heard the deadline change it had just been told. The evaporation didn't stop until we switched over to a version counter.
Now, even when a customer connects from a new device, even right after a server redeploy, the AI PM doesn't ask "where did we leave off last time?" That deadline-change request stays only in the window of the person in charge, and doesn't leak onto another team member's screen. But the changed deadline itself gets promoted to shared project information, so everyone sees the same schedule. Even after hundreds of turns pile up, that decision stays summarized and holds its spot at the front of the prompt.
As technology goes, it's unremarkable. But when it breaks, the customer has to re-explain the project from scratch every time, and as that fatigue builds, trust wavers. Just as the gap between a team that diligently keeps meeting notes and one that doesn't shows up months later as speed, how you handle conversation history is a service's baseline fitness.
The principles from the body still hold, with one addition: multiple servers share the same cache. The upside is that a cache one server updated is instantly visible to the others, which cuts down on the incident of each server holding its own stale copy. The cost is that the cache becomes one more external dependency, so you must keep the "cache miss = read the source" path alive, so that even if the cache goes down briefly, the system runs fine on the DB source of truth alone. In effect you design for a cache you can't trust as the default.
No. A lock is only safe when you keep the transaction short. A model call is a slow operation that takes several seconds, and holding the lock that whole time blocks every other write to the same conversation for exactly that long. The rule is simple: inside the locked section, do DB work only, and do slow things like model calls outside the lock.
Summarizing on the spot every time a request comes in slows the response, so you usually build them ahead of time as a background job. For example, each time the turns cross some threshold, you summarize the old stretch and store it separately, and when assembling the prompt you pull that stored summary. A summary is ultimately just another piece of data, so the same invalidation worry that comes with a cache follows it: when the source changes, you have to rebuild it.
A version lookup is a light operation that reads a single integer, but under extreme load even that burdens the DB. In that case you cache the version itself for a very short window (on the order of a few hundred milliseconds), or on write you push the new version straight into the cache to cut the read side's DB round trip. It's a trade that gives up a tiny bit of consistency to shed load, so use it only when the service can tolerate that brief lag.