03The network inside, and the next word

What the network adds after attention, how the next token is chosen, and what "it just predicts the next word" misses.

Where we are in the language modelRaw text, vectors, attention, tables and bends, next word; the current stage is highlighted.Raw textcut into tokensVectorschapter 1Attentionchapter 2Tables and bendschapter 3Next wordchapter 3

Attention gathers, the network adds what it knows

After attention, "anto" is [1.0, 0.4, 0.0]: a language, being spoken. It now passes through the classic network of the layer, a small network of exactly the kind described in the first article: a table, a bend, another table. Attention has gathered context from the sentence; the network adds what was learned during training about what usually follows what.

The toy network computing the correction for antoTwo neurons in layer 1, a correction table in layer 2; the correction is added to anto's numbers. Layer 1: two neurons read the numbers of "anto" languagespeakingpersonbiastotalafter the bend "anto" 1.00.40.0 Neuron A +10−1−0.6 0.4 0.4 fires Neuron B 00+1−0.5 −0.5 0 silent A: 1.0×1 + 0.4×0 + 0.0×(−1) − 0.6 = 0.4, positive, passed on. B: 0 + 0 + 0 − 0.5 = −0.5, negative, silenced. Layer 2: turn the neurons' outputs into a correction from Afrom Bcorrection outputs 0.40 language−10−0.4 speaking+1+0.5+0.4 person0−0.50.0 "anto" after attention[1.0, 0.4, 0.0] + correction [−0.4, +0.4, 0.0] = [0.6, 0.8, 0.0] Toy weights set by hand to behave like trained ones. Same tables, same multiply-and-add as in the first article.
learnedcomputed The classic network of our toy layer: two neurons, then a table that turns their outputs into a correction.

Read in plain words:

  • Neuron A fires when a token scores high on "language" and not on "person". It does here, at 0.4. In layer 2, its output lowers "language" and raises "speaking" by the same amount. Its effect amounts to a rule learned from experience: after a language has been named, what comes next is usually about how it is spoken, not another language.
  • Neuron B stays silent. It reacts to "person", which "anto" lacks, so it contributes nothing here. In another sentence, it would.
  • The network does not know which word it is processing. It reacts to numbers. Give it "truth" after attention, [0.3, 1.4, 0.0], and neuron A's total is 0.3 − 0.6, negative, so it stays silent too: no correction, because "truth" is not a language that has just been named.

Where do these weights come from? The same story as the "looking for" table. Nobody wrote "after a language, raise speaking". Whenever predicting what follows a language went wrong, the correction travelled back to these weights and nudged them; across millions of sentences, this is what remained. We set ours by hand to behave like trained ones. In a real model, the classic network of each layer holds tens of thousands of neurons, most of the model's parameters sit in these networks, and research suggests this is where much of its knowledge is stored[10].

Try it. In the companion spreadsheet, on the "Next token" tab, lower neuron A's bias from −0.6 to −1.2. Neuron A falls silent, there is no correction, and the model hesitates between "English" and "fluently", at 36% each.

Where "anto" stands at each stage

StageNumbers of "anto"Closest token in the table
From the table[0.5, 0.1, 0.0]"Esper": a fragment
After attention[1.0, 0.4, 0.0]"English": a language
After the network[0.6, 0.8, 0.0]"fluently": how a language is spoken

Each stage recomputes the numbers. Nothing moves: the token is simply described a little more precisely each time. The closest token is plain geometric distance in the three columns.

Choosing the next token

The final numbers of "anto", [0.6, 0.8, 0.0], are now compared with every row of the table, using the same match as before: multiply column by column, add. For "fluently": 0.6 × 0.7 + 0.8 × 0.8 + 0.0 × 0.0 = 1.06. The scores are then turned into probabilities with the same softmax recipe as in chapter 2:

Next token scores in the toy modelThe final vector of anto compared with each token: fluently 76 percent, speak 15, English 5, French 2, loudly 1.TokenMatchProbability of coming next␣fluently1.0676%␣speak0.9015%␣English0.785%␣French0.702%␣loudly0.641%7 others0.08 to 0.52less than 1% togetherMatch for fluently: 0.6 × 0.7 + 0.8 × 0.8 + 0.0 × 0.0 = 1.06Final vector of anto: [0.6, 0.8, 0.0]
computed Toy model. The percentages come from running the twelve-token model; they can be reproduced in a spreadsheet.

"fluently" wins with 76%. It won only because of what "anto" gathered from its neighbours through attention, and what the classic network knew about what follows a spoken language. Neither the token nor the table alone could have produced that answer.

From the last vector to the next wordThe final vector is scored against every token; one is picked and appended, and the loop starts again.Final vector of "anto"[0.6, 0.8, 0.0]Score every tokenin the vocabulary␣fluently76%␣speak15%␣English5%␣French2%␣loudly1%Pick one, append itI␣speak␣Esperanto␣fluently↻ the longer text goes through the model again to predict the next token
computed Toy model values from this section.

One token is then picked. Not always the most likely one: a controlled dose of randomness is usually added, which is why asking the same question twice can produce two different answers. The chosen token is appended to the text, and the whole process starts again to produce the next one. That is why long answers take time, and why vendors bill for every token the model writes. This is an LLM: a transformer trained to predict the next token, run in a loop.

From toy to real

Everything above is what a real model does. What changes is the scale. Here is our toy next to Meta's Llama 3 405B, one of the few large models whose architecture is fully published[7]:

Toy modelLlama 3 405B
Tokens in the vocabulary12128,000
Numbers per token316,384
Position signal added to each tokennoneyes
Attention heads per layer1128
Neurons in each classic network253,248
Layers (attention + network)1126
Longest text it can read4 tokensabout 128,000 tokens
Parameters (all weights)50405 billion

Every row is the same mechanism, repeated: look up, add the position, match, share, blend, add, correct, compare. A real model does it with 16,384 numbers instead of 3, 128 heads at once instead of one, and 126 times in a row, then compares the result with 128,000 tokens instead of 12. Our toy's 50 parameters are the 36 numbers of its table and the 14 weights of its network. For today's largest models, estimates run into the trillions; the third article explains what that means in hardware.

All the numbers, and who sets them

In our toy modelKindWho or what sets it
How tokens are cutchosenBuilt once by the tokenizer, from word frequencies, before training
Size of the tables: 12 tokens, 3 columnschosenEngineers (here, us)
Number of heads and layerschosenEngineers
Sharpness factor (×10)chosenEngineers (real models divide by √d instead)
The vocabulary tablelearnedTraining (here, written by hand to be readable)
"Looking for", "offers" and "hands over" tableslearnedTraining
The table merging the headslearnedTraining
Weights of neurons A and B, and the correction tablelearnedTraining
Matches, shares, blendscomputedRecomputed for every sentence, never stored
The numbers of "anto" after each stagecomputedRecomputed for every sentence
Next-word probabilitiescomputedRecomputed for every new token

"It just predicts the next word"

The phrase is true, and deeply misleading. Predicting the next word of any text, well, is an extremely demanding task. To finish the sentence "The capital of Australia is", the model needs a fact. To continue a legal argument, it needs to follow its logic. To predict the last line of a detective novel, where the culprit is named, it would in principle need to have followed the whole plot. The simplicity of the task hides how much the model has to capture in order to do it well.

But the phrase also contains a warning. The model produces the most plausible continuation, not a verified one. When it lacks the right information, the most plausible continuation can be a confident, well-formed and entirely invented answer. These "hallucinations" are not a bug that will simply be fixed: they are a direct consequence of how the model works, which is why the systems built around it need ways to check, ground and verify what it says.