01From words to vectors
How text is cut into tokens, how each token becomes a row of numbers, and where those numbers come from.
From a spam filter to a language model
The first article in this series gave us everything a neural network is made of: neurons that multiply, add and bend; layers that are tables of weights; and a training loop that adjusts those weights from examples. A language model like Claude or ChatGPT is built from exactly these pieces. But the spam filter had it easy in three ways, and a language model has none of them.
- Its inputs were prepared for it. The spam neuron never read an email. A small program read it, and handed the neuron five numbers, 1 or 0, one per clue. A language model receives raw text: words, in any order, of any length. Before any neuron can work, the text has to become numbers. That is the subject of this section.
- Each clue meant the same thing every time. "Contains a link" is a link, whatever else the email says. In language, the meaning of a word depends on the words around it: "speak" means one thing in "I speak French", another in "I speak the truth". The words must be able to inform one another before the neurons process them. That is what attention does (chapter 2).
- It answered one question with one score. Spam or not, a single number between 0 and 1. A language model answers a different question, again and again: of all the words it knows, which comes next? Its output is a probability for every word in its vocabulary (chapter 3).
From here on, we follow a single sentence, "I speak Esperanto", through a toy language model small enough to compute by hand. Its tables are tiny, but they are real tables, and everything it does is the multiplying, adding and bending you already know.
Tokens and IDs
The text is first cut into pieces called tokens. A token is often a whole word, but rare words are cut into several pieces. In "I speak Esperanto", the words "I" and "speak" are common enough to be tokens of their own, but "Esperanto" is cut into "Esper" and "anto". Notice that the space travels with the word that follows it: what the model receives is "I", " speak", " Esper", "anto".
Our toy model knows only twelve tokens. Each one has a number, its ID, like a line number in a list, so "I speak Esperanto" becomes 0, 1, 2, 3. From here on, the model never sees letters again, only numbers.
Who decides where to cut?
Not the model. The cutting is done by a separate and much simpler tool, the tokenizer, built before the model is trained. It knows nothing about meaning; it works purely on frequency.
The most common method, byte-pair encoding[1][2], starts from a large sample of text split into single characters. It finds the pair of neighbours that appears most often, say "e" followed by "a", and fuses it into a new piece. Then it does it again, and again, thousands of times, until the vocabulary reaches its target size, typically between 30,000 and 200,000 pieces.
Frequent words like "speak" end up as a single token; rare words like "Esperanto" stay in pieces, because the merges that would complete them never rank high enough. Once built, the tokenizer is frozen: the same text is always cut the same way. That is also why the cuts ignore grammar. "Esper" and "anto" are not meaningful parts of the word, just statistically convenient ones.
Two practical consequences. First, nothing is ever unknown: any word can be built from smaller pieces, down to single characters if needed, so the model can always handle a new brand name, a typo or a product code. Second, some simple tasks become surprisingly hard. The model never sees letters, only chunks of words, so counting the letters of a word, reversing it, or writing a text of exactly a given number of characters means working from information it does not directly have. The classic example, asking how many "r"s there are in "strawberry", tripped up many models when it became famous in 2024. Recent models usually get it right, often by spelling the word out letter by letter before counting, a workaround they have learned. The limitation has not disappeared; models have learned to compensate for it, more or less reliably depending on the model and the task.
A table of vectors
Alongside the vocabulary, the model keeps a table with one row per token. Each row holds a few numbers: in the vocabulary of the first article, a vector. In our toy model, each vector has three numbers, so the table has three columns. They have no names: just Column 1, Column 2 and Column 3.
Looking up
Turning a token into a vector is just a lookup: take the ID, go to that row, copy its three numbers. "speak" becomes [0.3, 0.9, 0.0]. There is no calculation and no intelligence involved. A real model also adds a few numbers that encode each token's position in the sentence, so it knows the order of the words; we leave them out to keep the toy simple.
Where the numbers come from
Before training, every box of this table holds a random number. Training then adjusts them with the same loop as the spam neuron: guess, measure the error, adjust. The third article explains where the answers come from for a language model. What matters here is the effect of those adjustments on the table.
Training never learns what a language is. What it discovers is that "French", "English" and "German" can replace one another in millions of sentences: after "I speak", before "fluently", next to "translate into". Every correction that helps the model predict what follows "French" also helps it with "English", so the two rows get pushed in the same direction, again and again. In the end, they hold similar numbers. Tokens that keep the same company end up with similar rows.
Naming the columns, after the fact
Now look at the finished table and ask a simple question: which tokens score high in each column? In Column 1, they are "Esper", "French", "English" and "fluently": tokens to do with languages. In Column 2, "speak", "truth", "fluently" and "loudly": tokens to do with speaking. In Column 3, "I" and "you": persons. So we give the columns nicknames: "language", "speaking", "person".
The order matters, and the nicknames come next. They describe what we observe in the numbers; nobody put them there. From here on we will use them, because "the language column" reads better than "Column 1", but they remain labels stuck on afterwards.
One caveat. In our toy, each column lines up neatly with one idea, because we chose the numbers to make the example readable. In a real model, with thousands of columns, it is rarely that tidy: an idea such as "language" is usually spread across many columns at once, and a single column mixes several ideas. Researchers can still find directions in a trained model that match ideas a human would recognise, and studying them has become a field of its own, called interpretability[3]. In these models, the meaning is never designed in advance: it can only be found after training.
What the numbers say
Three numbers per token make each token a point in a three-dimensional space. To draw it on a flat screen or page, we keep only two of the three columns, "language" and "speaking", and set the third aside:
Tokens used in similar ways land close together: "French" and "English" are neighbours, and so are "truth" and "loudly". And "speak" sits on its own, between the two groups. Its row is the same whether the sentence ends in "French" or in "the truth": the table alone cannot know which meaning is meant. "Esper" and "anto", the two pieces of a rare word, are weak on both columns. Fixing all this is the job of attention, in the next section.
Observed, not guaranteed
Nothing in this method guarantees that useful meanings will appear. Researchers set up the conditions (a large table, a lot of text, a rule for adjusting the numbers), let training run, and then examined what had come out. The first time this was done on a large scale, in 2013, the result was striking: words had organised themselves by meaning, without anyone telling them how[4]. The same approach has been repeated at every new scale since, and the result has held each time. Even the "scaling laws", which predict how much better a model gets with more data and computing power, are regularities found by experiment, not mathematical theorems[5].
As with the spam neuron (first article, chapter 3), this is something observed and measured after the fact, not something that can be proven in advance.