06From vector to next word: the LLM

Section 5 ended with a set of probabilities: after "I speak Esperanto", our toy model gave "fluently" 76%, "speak" 15%, "English" 5%, and almost nothing to the rest. A real model does exactly this over its whole vocabulary, tens of thousands of tokens, most of which get almost no chance at all.

One token is then picked. Not always the most likely one: a controlled dose of randomness is usually added, which is why asking the same question twice can produce two different answers. The chosen token is appended to the text, and the whole process starts again to produce the next one. Every word of an answer is the result of a full trip through the stack. That is why long answers take time, and why vendors bill for every token the model writes.

This is an LLM: a transformer trained to predict the next token, run in a loop.

"It just predicts the next word"

The phrase is true, and deeply misleading. Predicting the next word of any text, well, is an extremely demanding task. To finish the sentence "The capital of Australia is", the model needs a fact. To continue a legal argument, it needs to follow its logic. To predict the last line of a detective novel, where the culprit is named, it would in principle need to have followed the whole plot. The simplicity of the task hides how much the model has to capture in order to do it well.

But the phrase also contains a warning. The model produces the most plausible continuation, not a verified one. When it lacks the right information, the most plausible continuation can be a confident, well-formed and entirely invented answer. These "hallucinations" are not a bug that will simply be fixed: they are a direct consequence of how the model works, which is why the systems built around it need ways to check, ground and verify what it says.

What "a trillion parameters" means

A parameter is any number in the model that gets adjusted during training. Our toy table from chapter 02 holds 36 of them: twelve rows of three numbers. Look back at the classic network in chapter 04: every line connecting two neurons is another parameter, and that small drawing has 55. Now picture the real thing: a vocabulary table of tens of thousands of rows, each thousands of numbers long; every number used to form queries, keys and values in every attention head; every connection in every classic network; all multiplied by the number of layers. The count climbs fast: GPT-2, in 2019, had 1.5 billion parameters[17]; GPT-3, a year later, 175 billion[14]. For today's leading models, the figures circulating in the press reach into the trillions.

Three things are worth knowing about these numbers:

• Most are estimates. The leading vendors no longer publish parameter counts for their frontier models. Figures quoted in the press are usually informed guesses.
• Not all parameters work at once. Some recent models are built as a set of specialised sub-networks, called "experts", and only a few are used for each token. DeepSeek-V3, for example, has 671 billion parameters in total but uses about 37 billion for each token[18]. "Bigger" therefore no longer translates directly into "slower".
• Parameters are physical. Each one is a number that must sit in memory while the model runs. A trillion parameters take up roughly two terabytes, far more than any single chip holds, which is why serving these models requires clusters of specialised hardware, and why running a frontier model on your own premises is a very different proposition from running a small one.

More parameters give a model more capacity, but size alone does not make it better: the quality of the training data and of the training itself matter at least as much. Where that data comes from, and how those billions of numbers get their values, is the subject of the next chapter.