04What the model reads: context and memory

What the model actually receives with your question, why it remembers nothing, and what saved memory really is.

Your question never travels alone

Type "Who was the first president of the French Fifth Republic?" into an assistant, and you might think the model receives those words. It does not. The application in front of the model (the chat window, the company tool, the programming interface) first assembles one long text, and your question is only its last line:

What the model actually readsThe application assembles one long text: instructions, saved memories, documents, the conversation so far, and the new question at the end. In the new question, he looks back to Charles de Gaulle. What the model reads: one long textWhere it comes from Instructions"You are a helpful assistant. Answer concisely..." Saved memories"The user prefers short answers, in English." Documents (if any)attached files, or passages retrieved by a RAG system Conversation so far You: Who was the first president of the French Fifth Republic? Assistant: Charles de Gaulle, from 1959. Your new questionAnd how long did he stay in office? attention: "he" = "Charles de Gaulle" The vendor or theapplication Notes the applicationstored earlier You, or a search incompany documents Re-sent in fullwith every newquestion You The whole box is the context. The model continues it, one token at a time.
The context: everything the application assembles before the model reads your question.

Everything in this box is called the context. The model does exactly what chapters 1 to 3 described: it reads the whole text, attention links every token to the ones before it, and it predicts the next token, again and again. In "And how long did he stay in office?", the word "he" can only be understood because "Charles de Gaulle" is a few lines earlier in the same text. Attention makes the connection, exactly as "anto" connected to "Esper".

The model itself remembers nothing

This is the most counterintuitive point. The model's parameters were fixed at the end of training, and using the model does not change them. Between your first question and your second, nothing is kept inside the model. If the assistant seems to remember the conversation, it is because the application sends the entire conversation again, every time, with your new question at the end. Every exchange starts from a blank model reading a longer text.

Two kinds of knowledge

Each of these three questions draws on a different source:

  • "Who was the first president of the French Fifth Republic?" The answer, Charles de Gaulle, comes from the parameters. It was learned during training from countless texts that mention him. That knowledge is vast, but frozen and dated: the model knows nothing that happened after its training data was collected.
  • "And how long did he stay in office?" The model knows who "he" is only because the previous exchange is in the context. Start a new conversation with that question alone, and it has no way of knowing.
  • "What did our board decide last week?" That information is in neither. Unless the application adds the minutes to the context, the model can only produce the most plausible continuation, which is how confident, invented answers arise. Give it the minutes, and it can answer from them.

The parameters are what a person learned at school: vast, but fixed and dated. The context is the papers on their desk right now: precise and up to date, but limited in space.

What "memory" really is

Some assistants now "remember" you from one conversation to the next: your name, your preferences, your projects. No learning is involved. The application keeps notes, and slips them into the context of the next conversation, in the "saved memories" block above. It is a notebook the person rereads before each meeting. The notes can be shown, edited or deleted, and they take up space on the desk like everything else.

Retrieval-augmented generation (RAG), which companies use to make an assistant answer from their own documents, works on the same principle. A search engine finds the relevant passages and adds them to the context before the question. The model does not learn the documents; it reads them, each time.

The desk has a size, and a cost

Since the context is just text read by the model, everything in chapters 1 to 3 applies to it:

  • It has a maximum length, the context window: about 128,000 tokens for Llama 3.1, for example. Beyond that, something has to be left out, usually the oldest part of the conversation.
  • Length costs computation. Every token is compared with every token before it, so a context twice as long takes roughly four times the work.
  • Length costs money. Vendors bill per token read as well as per token written, and the whole re-sent conversation is read again with every question.
  • Length can cost reliability. In long contexts, models use information placed at the beginning or the end better than information buried in the middle[11].

Hence two practical habits. Start a new conversation when the subject changes, rather than letting one grow indefinitely. And when an answer must rest on specific facts, put those facts in the context rather than relying on what the model learned in training.