05Learning: where each adjustment comes from
The training loop on the toy model, with its numbers, and where each adjustment comes from.
The first article showed the training loop on a spam neuron. Here is the same loop on our toy language model from the second article, with its numbers. The only new ingredient is the answer key, which now comes from the text itself.
Go back to the moment training began. Suppose the classic network of the toy model (second article, chapter 3) has not yet learned its correction, so "anto" reaches the top of the model as [1.0, 0.4, 0.0], the value it had after attention. The training text says the next token is "fluently". The model's guess: "English" 36%, "fluently" 36%, "French" 25%, everything else 3%. It is undecided, and gives the right answer only about a third of the probability.
Score the error
The error depends on one thing only: how much probability the model gave to the right answer. 100% means no error; the lower the figure, the larger the error.
Find the direction
For the final vector, the direction has a remarkably simple form: towards the right answer, away from what the model expected. "What the model expected" is the average of all the token rows, each weighted by the probability the model gave it. Here that average is about [0.82, 0.46, 0.06], a point somewhere between "English", "French" and "fluently". The direction is the difference: fluently [0.7, 0.8, 0.0] minus expected [0.82, 0.46, 0.06], which gives [−0.12, +0.34, −0.06]. In plain words: less "language", more "speaking".
Take a small step
The vector moves a fraction of the way along that direction. With a step of one half, it becomes [0.94, 0.57, −0.03], and "fluently" rises from 36% to 61%. The model is not right yet, but it is less wrong.
But the final vector is not a knob
Nobody can set it directly: it is computed from the vocabulary table, the attention tables and the network's weights. As the first article explained, backpropagation carries the correction backward through everything that produced it[6], so that every parameter in the model gets its own small direction from that one wrong guess.
That includes the vocabulary table. The row for "fluently" is nudged slightly towards the vector that should have predicted it. The rows for "English" and "French" are nudged slightly away, because they got probability they did not deserve. This is how tokens that appear in the same contexts end up with similar rows, as the second article showed.
Why billions of tiny nudges and not a few big ones
A single sentence is a poor guide. In another training sentence, "I speak Esperanto" is followed by "and French", and the direction there would point the other way. So the model never takes a step on one example alone. Each step is computed from the average of all the directions in a batch, millions of tokens. Nudges that contradict each other cancel out; those that agree across many sentences accumulate. Over a million steps, what survives is what holds true across the text as a whole. The size of each step is a setting chosen by the engineers. Too large, and training becomes unstable; too small, and it never ends. It is tuned by trial and error, which is one more reason training is an empirical craft as much as a science.
The procedure as a whole is gradient descent. Picture a hiker lost in fog on a mountainside: unable to see the valley, they can still reach it by always stepping in the steepest downhill direction under their feet. Engineers may prefer another image: a control loop, applied to the parameters instead of to a motor or a valve. In practice the raw gradient is not applied directly: optimisers such as Adam[16] adapt the step size for each parameter, and the steps are tiny, a fraction of a thousandth.