How LLMs Remember Things: Context, Tools, and RAG Without the Magic
In the last article, I opened the hood on the core loop: tokenize the text, look back with attention, predict the next token, sample one, repeat.
That still holds.
But if you use these systems for more than ten minutes, you hit the next question.
Where did the answer get its facts?
You ask about a PDF and it quotes the PDF. You ask about today’s price and it somehow has today’s price. You ask a coding agent what a function does and it reads files that definitely were not in training. Then, in the same conversation, the same model forgets a detail you gave it a few messages ago.
That feels like one kind of memory from the outside. It isn’t.
The better question is not “does the model know this?” The better question is:
Where did this information enter the context?
The five places information can live
Here is the map I wish every AI product showed somewhere in its UI.
An answer can lean on trained weights. That is the baked-in stuff from training: language, common facts, code patterns, reasoning habits, all compressed into model parameters.
It can lean on the current context. That means the text visible right now: system instructions, chat history, pasted files, retrieved chunks, tool results, and the answer as it is being written.
It can lean on retrieved documents. Search outside the model, find useful chunks, paste them into context. This is common enough to have a name, retrieval-augmented generation, or RAG, and its own section later.
It can lean on tools. A tool might read a file, call an API, query a database, run a calculator, or check a calendar. The model asks. The app does the work. The result comes back as text or JSON.
And sometimes it leans on app memory: user preferences or facts stored by the product around the model. “Henry likes short answers for this kind of thing.” That sort of note.
By the time the model writes, most of those sources look the same to it: text on the desk. The source still matters, because each one fails differently. Weights go stale. Context overflows. Retrieval grabs the wrong chunk. Tools time out. Saved memory can be wrong in that especially annoying way where it sounds personal and therefore convincing.
Play with that difference directly:
Can I deploy the EU checkout change today?
General pattern: companies often protect payment systems with change freezes around holidays.
Conversation: the change is for EU checkout and affects card authorization.
Runbook chunk: EU checkout has a production freeze from Dec 18 through Jan 3. Emergency deploys need SRE approval.
Change calendar API: today is Dec 20. Change request CR-2047 is open but not approved.
Saved preference: answer operational questions with a clear go/no-go and the next action.
Grounded operational answer
No. Today is inside the EU checkout freeze and CR-2047 is not approved. Treat it as blocked unless SRE marks it as an emergency deploy.
The model can combine policy, live calendar state, and your preferred answer style.
The answer changes because the visible evidence changes. The model core is still just predicting tokens, but the surrounding system decides which memory surfaces get placed in front of it.
Nothing inside the model changed there. You only changed what it could see.
That is most of the trick.
Weights are compressed memory
When people say “the model knows Paris is the capital of France,” they usually imagine a database row hiding somewhere inside the model:
France -> ParisNice picture. Wrong picture.
During training, the model sees huge amounts of text and adjusts internal numbers until it gets better at predicting the next token. Those numbers hold patterns. Some are grammar. Some are style. Some look a lot like facts. This is the part the first article walks through in detail, if you want to see how that adjustment actually works.
The original RAG paper uses a helpful split: parametric memory and non-parametric memory. Parametric memory is what is baked into the model parameters. Non-parametric memory is an explicit external store, such as a search index over Wikipedia or your company’s docs.
That split is not trivia. If a fact lives in the weights, you can’t cleanly inspect where it came from, update only that one fact, or ask for a source. If the fact lives in a document store, you can edit the document and re-index it.
Weights are great for broad sense-making. They are bad for fresh prices, today’s deploy calendar, your private runbook, or the name of the server in your basement.
Please don’t put my basement server name in the weights. Put it somewhere I can delete it.
Context is not memory; it is the desk
The context window is the text the model can see while generating a response. Anthropic’s context window docs describe it as working memory, separate from the training corpus.
I like the desk metaphor better.
Whatever is on the desk can be used right now. Whatever is not on the desk may as well be in another room. A bigger desk helps, sure. It also gives you more room to bury the one paragraph that mattered.
Everything on the input side shares this space:
- system instructions
- developer instructions
- tool definitions
- the conversation history
- retrieved document chunks
- tool results
- images or file text
The answer the model writes is not on this list, and that is the point. While it is being generated, the output is its own separate space, not part of the input on the desk. It has its own limit too, the max output tokens, which caps how long a single reply can get.
The output only lands on the desk on the next turn. Once the reply is finished, it gets appended to the conversation history, and now it is plain input. A long answer you loved a moment ago becomes one more thing taking up room, competing for space with everything above.
Tiny desk, on purpose:
Be helpful, cite policy snippets, and ask before risky production actions.
searchDocs(query), getChangeCalendar(service), fetchIncident(id).
Can I deploy the EU checkout change today?
The user is asking about CR-2047 for EU checkout card authorization.
EU checkout freeze Dec 18-Jan 3. Emergency deploys require SRE approval.
Today Dec 20. CR-2047 status: open. SRE approval: false.
Earlier brainstorming about rollout naming, monitoring dashboards, and release notes.
Real context windows are much larger than this toy budget, but the trade-off is the same: instructions, history, retrieved chunks, and tool results all compete for one finite input space. Turn on compaction and watch the total drop as verbose material is swapped for summaries, which is exactly how you fit more of what matters under the same ceiling.
Real context windows are much larger. Same problem.
This is where I think a lot of AI advice gets the order wrong. People obsess over prompt phrasing. Phrasing matters, but the bigger question is usually: did the right evidence even make it onto the desk?
The model can look like it forgot something without deleting anything. The fact may still exist in a database, an old chat, a file, or a vector index. It just was not placed back in view.
Tools are not magic hands
Tool use sounds spooky from the outside. The model checked the weather. The model searched the docs. The model opened a file.
The less spooky version:
- The application tells the model which tools exist and what arguments they accept.
- The model emits a structured request to use one of those tools.
- The application decides whether to execute that request.
- The application sends the tool result back to the model.
- The model continues generating with that result now in context.
OpenAI’s function calling docs describe this as a multi-step flow: provide tools, receive a tool call, execute code on the application side, send back the output, then receive a final response. The important phrase is “application side.” The model is not reaching through the screen with hands. It is asking the wrapper system to do something.
That wrapper can be tiny, like a weather API. It can also be dangerous, like a shell or deployment tool. The model can ask. The product decides what is allowed, what runs, what gets redacted, what gets logged, and what text returns.
For a while, every product wired up that wrapper by hand, in its own shape. These days there is a common plug for it: the Model Context Protocol, or MCP. Anthropic introduced it as an open standard, and it caught on fast, so Claude, ChatGPT, and editors like VS Code and Cursor now speak it. The idea is boring in the good way. You write an MCP server that exposes a set of tools once, and any MCP-capable app can call them, instead of every app reinventing its own tool plumbing. It does not change the five steps above. The model still just asks, and the application still decides. MCP is mostly agreement on how the tools get described and how the request and result travel between the app and whatever is on the other end.
Step through it:
Once a tool result comes back, it is context. A good result gives the model fresh, exact information it could never have learned during training. A bad result poisons the desk.
This is also why agents eat context so quickly. Tool definitions cost tokens. Tool results cost tokens. Notes from previous steps cost tokens. If an agent calls ten tools and keeps every result in full, it can drown in its own receipts.
The good ones know what to keep visible.
RAG is a searchable memory outside the model
RAG stands for retrieval-augmented generation. The name is a mouthful. The idea is plain:
- Put your documents in a search system.
- When the user asks a question, search for relevant pieces.
- Paste those pieces into the model’s context.
- Ask the model to answer using them.
The model does not absorb the whole knowledge base. It reads the handful of retrieved chunks that made it into the prompt.
There are two moments. Before anyone asks a question, you split documents into chunks, convert each chunk into an embedding, and store those embeddings. When the user asks, you embed the question too, search for nearby chunks, maybe add keyword search, maybe rerank, then put the best pieces on the desk.
An embedding is just a vector: a list of numbers. The useful mental image is a point in a huge meaning-space. The query becomes one point. Every document chunk becomes another point. Retrieval measures the distance from the query to each chunk and ranks the nearest ones first.
OpenAI’s embeddings docs describe embeddings as measuring relatedness between text strings, where smaller distance means higher relatedness. In current OpenAI embedding models those vectors are normalized to length 1, so cosine similarity can be computed as a plain dot product, and cosine similarity and Euclidean distance give the same ranking.
That still sounds abstract, so here is a tiny version you can push around. Real embeddings have hundreds of dimensions, so I flattened them onto a plane. Related text clusters. Drag the query and watch which chunks fall close:
The ring is the part people skip. Retrieval is not just “rank by distance.” It also cuts. You draw a cutoff around the query, keep what lands inside, and drop the rest. Tighten the ring and you may starve the model of context; widen it too far and you drag in noise. That surviving handful is the only text the model gets to read.
A query like “can I get my money back” can retrieve a document about “refund policy” even when the words barely overlap, because the query point lands close to the refund-policy point. OpenAI’s retrieval docs make the same point: semantic search can surface results with few or no shared keywords.
Embeddings are still soft meaning search. That is great when the user phrases the same idea differently from the document. It is weaker when the thing that matters is an exact code, product ID, error number, or legal phrase.
That is why serious retrieval systems often go hybrid. They use embeddings for meaning and lexical search, such as BM25, for exact words. Anthropic’s Contextual Retrieval write-up calls out this exact issue: embeddings can miss exact matches, while keyword methods can catch identifiers like an error code. In the demo, flip the exact-code query into hybrid mode and watch TS-999 win for exactly that reason.
RAG also depends heavily on chunking. If you split documents badly, you can separate a sentence from the heading that explains what it applies to. The retriever may find the sentence, but the model receives it without the context needed to use it. This is one reason “just make chunks smaller” and “just make chunks bigger” are both incomplete advice.
Small chunks are precise but can lose context. Big chunks keep context but bring noise. Good RAG is mostly taste and plumbing: enough surrounding text to make the evidence usable, not so much that the model has to rummage through a drawer full of receipts.
The RAG pipeline, end to end
Here is the whole pipeline as a little machine:
Split documents into chunks
EU checkout freeze Dec 18-Jan 3.
Emergency deploys require SRE approval.
Rollback owner is payments-oncall.
RAG is two systems glued together: an indexing/search system that chooses evidence and a model that writes from whatever evidence made it into the prompt.
Watch where the failures happen.
If the final answer is bad, the model may not be the first thing to blame. Maybe the document was never indexed. Maybe the chunk boundary cut off the title. Maybe the embedding search found semantically similar but operationally irrelevant text. Maybe the reranker promoted the wrong chunk. Maybe the prompt included six chunks when one would have been better. Maybe the answer instruction said “be concise” but the retrieved text needed nuance.
RAG failures often look like generation failures because generation is the visible part. But the answer can only be as grounded as the context it receives.
Retrieval is not “the model looking things up.” Retrieval is a separate system choosing what to put on the desk.
Once the chunks are in the prompt, the model still does the same old thing. It predicts the next token. It can quote, summarize, compare, and reason over the retrieved text because that text is visible. If the relevant chunk never arrives, the model may fall back to weights and produce something plausible.
That’s the classic RAG hallucination shape: the generator sounds fine, but the evidence never showed up.
Context management is the real product
At this point, the interesting part is no longer the model by itself. It is the system around the model.
A coding agent has to decide which files to read and which command output to keep. A support bot has to decide which policy chunks are relevant and whether today’s account state needs an API call. A writing assistant has to decide whether your saved tone preference matters for this paragraph. A long-running research assistant has to decide what to summarize and what to preserve verbatim.
Anthropic’s context engineering article frames one pattern as “just in time” context: keep lightweight references, then load the relevant data with tools at runtime. That is how humans work most of the time. I do not memorize my whole filesystem. I remember that there is probably a file, a branch, a note, a ticket, or a command that will get me the detail when I need it.
That also explains why there is no single winner between long context, RAG, and tools. And it points to a bias I hold, and have for a while: reach for RAG last, not first. If there is an easier way to get the model the same information, use it. It is tempting to treat “add a vector database” as the default answer to “the model needs to know things,” but every stage you add is a stage that can fail. Bad chunking splits a sentence from its heading. The retriever grabs something that reads similar but answers a different question. The reranker promotes the wrong chunk. And the failure is quiet: when the right chunk never arrives, the model does not stop, it guesses, and the guess sounds just as confident as a grounded answer. If you already know which document you need, a direct tool call that fetches it beats searching for it. If the whole thing fits on the desk, put it on the desk. That same Anthropic article makes the point in a pointed way: they dropped vector search from Claude Code in favor of plain agentic search, letting the model grep and read files the way a person would, and found it worked better than embedding everything up front.
None of that means RAG is dead. When the corpus genuinely dwarfs the context window, when you need provenance and citations, or when you are searching a large, mostly stable pile of unstructured prose, retrieval earns its keep. (If the data churns constantly, though, an index is the wrong tool. You spend your life re-chunking and re-embedding to stay current, and a live tool call would have just fetched the fresh version.) The point is narrower: RAG is a real system with real failure modes, so it should be a choice you make on purpose, not the reflex you start with.
The weights are not one of the choices here. You do not route a fact into them; that happened during training, long before your app existed. What you actually control is everything else, and the routing question is simple:
- Does this belong in the prompt? Yes, if it matters right now.
- Does this belong in RAG? Only if the corpus is too big for the desk and you cannot just fetch the right document directly.
- Does this belong behind a tool? Yes, if it is live, structured, private, or action-oriented.
- Does this belong in app memory? Maybe, if it is durable, user-approved, and useful across sessions.
Bad AI products blur these together. They let the model guess fresh facts from weights. They stuff enormous prompts with irrelevant context. They retrieve chunks without checking whether the chunks answer the question. They expose too many tools and make the model choose between ambiguous actions. They store memories because they can, not because the future user benefits.
Good AI products are boring in a very specific way. They make the source of truth obvious. They fetch only what is needed. They keep enough context to reason, but not so much that the model has to swim through clutter. And they work hard to get the model to say “I don’t know” when the desk does not contain the answer, because that is not the default. A model trained to predict the next token would rather produce a confident, plausible guess than admit a gap. Abstaining is a behavior you have to engineer in: instructions, grounding, sometimes a separate check. It does not come free.
The practical model
Here is the version I keep in my head:
Weights are long-term intuition. Broad, powerful, hard to inspect, slow to update.
Context is the current desk. If the text is there, the model can use it. If it is not, the model cannot directly see it.
RAG is the library index. It searches external documents and places the best-looking pages on the desk.
Tools are phone calls to the outside world. The model asks, the application calls, the result comes back as context.
App memory is a notebook the product controls. Useful sometimes. Creepy if done badly.
All of that still flows into the same next-token engine from the last article. That is the funny part. LLM systems can look like they remember, browse, reason, and act because the wrapper keeps changing the text the model sees before each prediction.
No hidden filing cabinet.
Just a very carefully managed desk.
The demos above use hand-authored data so they run entirely in your browser: no live model, no API calls, no retrieval backend. The vectors, token counts, tool traces, and distances are made up to teach the mechanics. The systems they show match the shape of real LLM apps, but the numbers are toy numbers.