02From words to vectors

A computer cannot work with words, only with numbers. So the first thing a transformer does is turn the text into numbers. The easiest way to see how is to build a model by hand. Ours will be tiny, but it works exactly like the real thing, and we will use it all the way through this article.

A vocabulary

Our toy model knows only twelve tokens. Each one has a number, its ID, like a line number in a list. When text comes in, the tokenizer (chapter 05) cuts it into tokens and replaces each one with its ID. "I speak Esperanto" becomes 0, 1, 2, 3.

A table of vectors

Alongside the vocabulary, the model keeps a table with one row per token. Each row holds a few numbers, and that row is the token's vector. In our toy, each vector has just three numbers. To make them readable, we have given each column a meaning: how much the token has to do with a language, with speaking, and with a person, from 0 (not at all) to 1 (very much).

Try it yourself. Change a number in the table and watch the rest of the model follow. Download the toy model as a spreadsheet, built with the help of Claude.

Looking up

Turning a token into a vector is nothing more than a lookup: take the ID, go to that row, copy its numbers. ID 1 gives [0.3, 0.9, 0.0], so "speak" is a little about a language, very much about speaking, and not about a person. Do the same for the other three IDs and the sentence becomes four short lists of numbers. That is the entire step. There is no calculation and no intelligence involved, just a table lookup.

What the numbers say

Three numbers can be read as coordinates, which makes each token a point in space. If we plot just the first two columns, the vocabulary turns into a map:

Tokens that are used in similar ways land close together: "French" and "English" are neighbours, and so are "truth" and "loudly". Three details matter for what follows:

• "speak" sits on its own, between both groups. Its vector is the same whether the sentence ends in "French" or in "the truth". The table cannot know which meaning is meant.
• "Esper" and "anto" are weak. Each piece leans a little towards "language", but neither is anywhere near "French". On its own, a fragment means little.
• Nothing here depends on the sentence. A token always gets the same row, whatever surrounds it.

Fixing those weaknesses is the job of attention. In chapter 05 we will do it with these same numbers and watch "anto" move across the map.

From toy to real

A real model differs from ours in size, not in principle:

• Size. A real vocabulary has 50,000 to 200,000 rows, and each row holds several thousand numbers. A table of 100,000 rows by 4,000 numbers is already 400 million numbers, and these are parameters, before the model has done anything else.
• Position. A second set of numbers, encoding each token's place in the sentence, is added to its vector so the model knows the order of the words. We leave it out of the toy to keep the numbers simple.

What does not change is the mechanism: ID, row, numbers.

Who fills in the table?

In our toy, we chose the columns and wrote the numbers ourselves. In a real model, nobody does either. Engineers decide only the size of the table: one row per token, and a fixed number of columns. Every box starts as a random number.

Training then does the rest. As we will see in chapter 07, the model learns by guessing the next token over and over, and after each wrong guess every number involved is nudged slightly in the direction that would have helped. The numbers in this table are nudged along with all the others.

Here is why meaning emerges. "French" and "English" appear in the same surroundings: after "I speak", before "fluently", next to "translate into". Whenever the model stumbles on such a sentence, the same kind of correction helps both, so their rows get pushed the same way. Over millions of sentences, tokens that keep the same company drift together on the map. As the linguist J. R. Firth put it in 1957, "You shall know a word by the company it keeps"[6].

Which brings us back to our labels. Since nobody tells the model what the columns should mean, it has no reason to line them up with human ideas such as "language" or "person". Any arrangement that helps predict the next token works equally well, so meaning ends up spread across many columns at once. Researchers can still find directions in a trained model that match ideas a human would recognise, and studying them has become a field of its own, called interpretability[7]. But those directions have to be discovered after training; they are never designed in advance. Our labels are training wheels.

Nothing in this method guarantees that useful meanings will appear. Researchers set up the conditions (a large table, a lot of text, a rule for nudging the numbers), let training run, and then checked what had come out. The first time this was done on a large scale, in 2013, the result was striking: words had organised themselves by meaning, without anyone telling them how[8]. The same bet has been placed again at every new scale since, and so far it has kept paying off. But it remains something we observe and measure after the fact, not something we can prove in advance. Even the famous "scaling laws", which predict how much better a model gets with more data and computing power, are observed regularities, not theorems[9].