02Attention: letting the words inform each other
How each word draws meaning from the words before it, worked through by hand, and why long texts cost more.
The problem
At the end of chapter 1, every token has its row of numbers, and that row never changes, whatever the sentence. "speak" has the same vector in "I speak French" and in "I speak the truth". "Esper" and "anto" are fragments that mean little on their own. To understand a sentence, each token's numbers have to take its neighbours into account.
The idea, in one sentence
Attention does this in one simple way: each token adds to its own numbers a weighted average of the numbers of the tokens before it. The weights, which we will call shares, depend on how relevant each earlier token is. Relevant tokens get a large share, irrelevant ones almost nothing.
Why only the tokens before it? Because these models write text one token at a time, from left to right: a token cannot consult tokens that have not been written yet. When the model reaches "speak", it cannot know whether "French" or "the truth" will follow. The ambiguity is resolved one step later, when "French" (or "truth") looks back at "speak".
Attention, by hand
Let's follow the last token of "I speak Esperanto", the fragment "anto", whose row is [0.5, 0.1, 0.0]. Before reading on: which of the four tokens do you think "anto" will pay most attention to?
Step 1: the match
The match between two tokens measures how much their rows point in the same direction. The rule: multiply the two rows column by column, then add the three products. Mathematicians call this the dot product of two vectors. Each column works like a vote: if both tokens score high on the same column, the product is large and the match goes up; if either one scores zero, that column contributes nothing. So two tokens match strongly when they are high in the same columns.
Step 2: the shares
The four matches must become shares that add up to 100%. Three operations, each with a reason:
- Magnify: multiply by 10. With only three columns, the matches are small and close together. Multiplying by 10 widens the gaps. We call this factor the sharpness: the higher it is, the more decisively attention picks a favourite.
- Exponential: raise the number e (2.718...) to that power. This turns each value into a positive number and amplifies the differences: a gap of 0.8 between "Esper" (3.2) and "speak" (2.4) becomes a ratio of more than two (24.53 against 11.02). The best matches pull ahead; the weak ones fade.
- Divide by the total. The four results add up to 50.12. Dividing each by 50.12 gives shares that add up to exactly 100%.
Why not just divide?
We could divide the matches by their total directly, but the result would be much flatter: "I" 1%, "speak" 29%, "Esper" 39%, "anto" 31%. Every token would get a sizable share. And in real models, matches can be negative, and a negative share makes no sense. The exponential solves both problems at once. This recipe, magnify, exponential, divide by the total, is called softmax, and it appears everywhere in language models: chapter 3 uses it again to turn scores into next-word probabilities.
Step 3: blend and add
"anto" now takes each earlier row, multiplies it by that row's share, adds them all up, and adds the result to its own row:
Three things to notice:
- "anto" looks mostly at "Esper" (49%). It is completing its own word. It takes a good share from "speak" (22%) and almost nothing from "I" (2%), which has little to contribute.
- The blend is added, not substituted. "anto" keeps its own numbers and gains what it collected from the others.
- The result has changed meaning. At [1.0, 0.4, 0.0], "anto" now scores high on the "language" column and somewhat on "speaking". A fragment near nothing in particular has become, in the model's numbers, a language being spoken.
The same arithmetic in "I speak the truth" pulls "truth" the other way. There, "truth" gives 85% of its share to "speak", and its numbers climb on the "speaking" column: something being said, not a language. Same table, same rule, different neighbours, different result. That is how the ambiguity of "speak" gets resolved.
What our toy language model simplifies: three versions of each token
A word that only looks for words like itself is a bit like a manager who only hires clones. In our toy, each token used its own row for everything: to look for relevant neighbours, and to be found by them. A row always matches best with rows like itself, which is why "anto" gave its largest shares to its other half, "Esper", and to itself. But very often, a word needs to find something different: a pronoun needs its noun, a language needs the verb that governs it.
Real models solve this by giving each token three versions of its row, each produced by passing the row through its own small table of weights, exactly like a layer in the first article:
- what the token is looking for (specialists call it the query);
- what the token offers, to be found by others (the key);
- what the token hands over once chosen (the value).
Let's give "anto" a "looking for" table and see what changes. To keep the numbers simple, the other two versions stay equal to the rows themselves.
With its own row, "anto" looked for tokens like itself and found "Esper". With the "looking for" table, it looks for a token that scores high on "speaking", finds "speak", the verb, and gives it 94% of its attention. The table holds a single meaningful weight, which reads as a rule: "a language looks for speaking".
Where does the "looking for" table come from?
We wrote ours by hand. In a real model, nobody does. Like the vocabulary table, it starts as random numbers and is adjusted by training. Whenever "anto" looked in the wrong place and the next word was predicted badly, the correction travels back through the shares to the "looking for" table, and nudges its weights towards whatever would have directed attention more usefully. In millions of sentences, a language name is followed by something about speaking it, so the weight linking "language" to "speaking" gets nudged up again and again. The rule is never written; it is what remains after training. Real tables are thousands of numbers wide, every head of every layer has its own, and like the columns of chapter 1, they rarely read as tidy rules. The "offers" and "hands over" tables are learned in exactly the same way.
Try it. In the companion spreadsheet, on the "Attention" tab, change the "looking for" table of head 2: set the 1 to 0 and put 1s on the diagonal instead, so that "anto" looks for tokens like itself again. It goes back to "Esper", exactly like head 1.
Looking for several things at once: attention heads
We now have a dilemma. Without a "looking for" table, "anto" completes its own word: 49% to "Esper". With one, it finds its verb: 94% to "speak". Both are useful, but one set of shares can only do one at a time. So a real model runs several attention calculations side by side, each with its own three tables. Each one is called a head. Here are our two calculations as two heads of the same layer:
Each head reads the same sentence from a different angle. Head 1 answers "which word am I part of?"; head 2 answers "which verb governs me?". The model does not have to choose: it places the two blends side by side and passes them through one more table, learned like the others, which merges them into the single update added to "anto". In our walkthrough, we keep only head 1, so the numbers stay easy to follow.
Real models work the same way with many more heads: 8 per layer in the original 2017 model[6], 128 in Llama 3 405B[7]. Nobody assigns their roles. Researchers who examine trained models find heads that link pronouns to their nouns, heads that look at the previous word, heads that connect verbs to their objects, and many whose role is hard to describe[8]. In "The trophy didn't fit in the suitcase because it was too big"[9], one head can link "it" to "trophy" while another links it to "was": two kinds of context, gathered at the same time.
Distance doesn't matter, length does
In the trophy sentence, "it" connects to "trophy", several words back, as easily as to its neighbour. This is exactly what recurrent networks struggled with: by the time they reached "it", their memory of "trophy" had faded. But attention has a cost. Every token must compute a match with every token before it, so doubling the length of a text roughly quadruples the work. This is why every model has a maximum length of text it can handle, its "context window", and why long documents cost more to process (chapter 4).