04Many angles, many layers

Attention heads side by side, a classic network inside every layer, and layers stacked on top of each other.

A single pass of attention is not enough to understand a sentence. Transformers repeat it in two ways: side by side, and on top of each other.

Side by side: attention heads

Within one layer, attention does not run once but several times in parallel, each run being called a "head". The original 2017 model had 8 heads per layer[5]; large modern models have dozens. Each head has its own learned way of forming queries, keys and values, so each ends up looking for something different. Researchers who examined trained models found heads that consistently link a pronoun to the noun it refers to, heads that simply look at the previous word, and heads that connect a verb to its object[11]. Nobody assigned these roles: they emerged during training because they help predict the next word. Many other heads do things that remain hard to describe.

Illustrative. Studies of trained models have found heads specialised in this way [11], but many heads remain hard to interpret.

After the heads have done their work, their findings are combined into a single update of each word's vector.

Between two attention steps: a classic network

Attention is only half of each layer. The other half will look familiar to anyone who has met neural networks before.

Within a layer, the "Network" boxes are one and the same network, with the same weights, applied to each word separately.

For readers who know neural networks

It is tempting to picture the transformer as a front end that prepares the input for a "real" neural network behind it. In models like ChatGPT, Claude or Gemini, that is not how it works: the classic network is inside every layer. Each layer alternates two steps. First attention, where words exchange information. Then a classic feed-forward network, applied to each word's vector separately, with the same weights for every word. The words talk, then each thinks on its own, then they talk again.

This classic part is far from minor: in a typical model, roughly two-thirds of all parameters sit in these feed-forward networks, and research suggests this is where much of the knowledge absorbed during training is stored, while attention handles the context[12].

The "front end" picture does apply elsewhere. Models such as BERT[13] are often used to turn text into vectors that a separate, smaller network then classifies, and the embedding models used in document search work in the same spirit.

On top of each other: layers

The original model stacked 6 such layers; GPT-3, in 2020, had 96[14]. Each layer does not replace a word's vector but adds to it, a little like readers successively annotating the margins of the same page: the original word is never lost, and each layer contributes its refinements. Studies of trained models suggest a rough division of labour, with early layers handling surface features such as word pieces and grammar, middle layers building meaning, and the last layers preparing the prediction. It's a tendency rather than a strict rule, but it captures why depth matters: understanding is built up gradually.