03What attention does to the vectors
How each word looks back at the words before it and takes on their context.
At the end of the previous chapter, every word had a vector, but each vector knew only about its own word. Attention is the step where words start taking their context into account.
The principle is simple: each word looks at the other words in the sentence, decides which ones matter to it, and blends a little of their meaning into its own vector. Words that matter a lot contribute a lot; words that don't matter contribute almost nothing.
How does a word decide what matters? Picture a meeting where everyone has three things: a question they want answered, a badge describing what they know, and the knowledge itself. Each participant compares their question with everyone's badge. The better the match, the more attention they pay to that person, and the more of that person's knowledge they take away. In the transformer, these three things are also vectors, derived from each word's own vector: the question is called the query, the badge the key, the knowledge the value.
One rule applies in the models behind ChatGPT, Claude or Gemini: a word can only look at the words before it. These models write text one word at a time, from left to right, so a word cannot consult words that haven't been written yet. This has a consequence for our example. When the model reaches "speak", it cannot know whether "French" or "the truth" will follow. The ambiguity is resolved one step later, when "French" looks back at "speak" and absorbs the fact that it is being spoken. It is the vector of "French" that ends up meaning "a language, being spoken", just as the vector of "truth" ends up meaning "something being told".
Attention works just as well when the clue comes first. In "the river bank" and "the investment bank", the word "bank" looks back at "river" or "investment" and its vector shifts towards water's edge or towards finance. And distance doesn't matter. In "The trophy didn't fit in the suitcase because it was too big"[10], the word "it" can connect directly to "trophy", several words back, as easily as to its neighbour. This is precisely what recurrent networks struggled with: by the time they reached "it", their memory of "trophy" had faded.
Who decides that "French" should look at "speak", or that "it" should look for a noun? Nobody. The way queries, keys and values are derived from each word is learned during training, like everything else in the model. The rules of grammar and reference are never written down; they emerge because they help predict the next word.
Section 5 works through this mechanism by hand, with the numbers of our toy model: you will be able to check every percentage with a calculator.
Attention has a cost. Since every word must be compared with every word before it, doubling the length of a text roughly quadruples the work. This is why models have a maximum text length they can handle, the "context window", and why very long documents are more expensive to process.