01From words to vectors

How text is cut into tokens, how each token becomes a row of numbers, and where those numbers come from.

Where we are in the language modelRaw text, vectors, attention, tables and bends, next word; the current stage is highlighted.Raw textcut into tokensVectorschapter 1Attentionchapter 2Tables and bendschapter 3Next wordchapter 3

From a spam filter to a language model

The first article in this series gave us everything a neural network is made of: neurons that multiply, add and bend; layers that are tables of weights; and a training loop that adjusts those weights from examples. A language model like Claude or ChatGPT is built from exactly these pieces. But the spam filter had it easy in three ways, and a language model has none of them.

  • Its inputs were prepared for it. The spam neuron never read an email. A small program read it, and handed the neuron five numbers, 1 or 0, one per clue. A language model receives raw text: words, in any order, of any length. Before any neuron can work, the text has to become numbers. That is the subject of this section.
  • Each clue meant the same thing every time. "Contains a link" is a link, whatever else the email says. In language, the meaning of a word depends on the words around it: "speak" means one thing in "I speak French", another in "I speak the truth". The words must be able to inform one another before the neurons process them. That is what attention does (chapter 2).
  • It answered one question with one score. Spam or not, a single number between 0 and 1. A language model answers a different question, again and again: of all the words it knows, which comes next? Its output is a probability for every word in its vocabulary (chapter 3).
From a spam filter to a language modelThe spam filter: prepared clues, tables and bends, one score. The language model: raw text, vectors, attention, the same kind of tables and bends repeated layer after layer, and a probability for every next word. The spam filter (first article) Cluesprepared by a program Tables and bendsneurons, layers One scorespam or not same building blocks The language model (this article) Raw textcut into tokens Vectorschapter 1 Attentionchapter 2 Tables and bendschapter 3 Next worda probability each repeated layer after layer already known (first article) new for language

From here on, we follow a single sentence, "I speak Esperanto", through a toy language model small enough to compute by hand. Its tables are tiny, but they are real tables, and everything it does is the multiplying, adding and bending you already know.

Tokens and IDs

The text is first cut into pieces called tokens. A token is often a whole word, but rare words are cut into several pieces. In "I speak Esperanto", the words "I" and "speak" are common enough to be tokens of their own, but "Esperanto" is cut into "Esper" and "anto". Notice that the space travels with the word that follows it: what the model receives is "I", " speak", " Esper", "anto".

Our toy model knows only twelve tokens. Each one has a number, its ID, like a line number in a list, so "I speak Esperanto" becomes 0, 1, 2, 3. From here on, the model never sees letters again, only numbers.

Who decides where to cut?

Not the model. The cutting is done by a separate and much simpler tool, the tokenizer, built before the model is trained. It knows nothing about meaning; it works purely on frequency.

The most common method, byte-pair encoding[1][2], starts from a large sample of text split into single characters. It finds the pair of neighbours that appears most often, say "e" followed by "a", and fuses it into a new piece. Then it does it again, and again, thousands of times, until the vocabulary reaches its target size, typically between 30,000 and 200,000 pieces.

How a tokenizer builds its vocabularyFrequent pairs are fused step by step until speak is one token; Esperanto stays in two pieces.Start: single charactersMost frequent pair: e + aNext: s + pNext: ea + kNext: sp + eakspeakspeakspeakspeakspeakOne tokenA rarer word stays in pieces␣EsperantoAmber: the pair fused at this step. Merge order is illustrative.
Byte-pair encoding: the most frequent neighbouring pieces are fused, again and again, until the vocabulary reaches its target size [1][2].

Frequent words like "speak" end up as a single token; rare words like "Esperanto" stay in pieces, because the merges that would complete them never rank high enough. Once built, the tokenizer is frozen: the same text is always cut the same way. That is also why the cuts ignore grammar. "Esper" and "anto" are not meaningful parts of the word, just statistically convenient ones.

Two practical consequences. First, nothing is ever unknown: any word can be built from smaller pieces, down to single characters if needed, so the model can always handle a new brand name, a typo or a product code. Second, some simple tasks become surprisingly hard. The model never sees letters, only chunks of words, so counting the letters of a word, reversing it, or writing a text of exactly a given number of characters means working from information it does not directly have. The classic example, asking how many "r"s there are in "strawberry", tripped up many models when it became famous in 2024. Recent models usually get it right, often by spelling the word out letter by letter before counting, a workaround they have learned. The limitation has not disappeared; models have learned to compensate for it, more or less reliably depending on the model and the task.

A table of vectors

Alongside the vocabulary, the model keeps a table with one row per token. Each row holds a few numbers: in the vocabulary of the first article, a vector. In our toy model, each vector has three numbers, so the table has three columns. They have no names: just Column 1, Column 2 and Column 3.

Toy model: a table of vectors with unnamed columnsTwelve tokens, three unnamed columns; nicknames are added afterwards by looking at which tokens score high in each column."I speak Esperanto"IID 0␣speakID 1␣EsperID 2antoID 3Each ID points to a row.The row's numbers are the vector.No calculation: just a lookup.IDTokenColumn 1Column 2Column 30I0.00.10.91␣speak0.30.90.02␣Esper0.60.20.03anto0.50.10.04␣French0.90.20.15␣English0.90.30.16␣truth0.00.60.07␣fluently0.70.80.08␣loudly0.00.80.09␣the0.10.10.110␣you0.00.10.911.0.00.10.0Who scores high?Esper, French,English, fluentlyspeak, truth,fluently, loudlyI, youNickname, given afterwards"language""speaking""person"
learnedchosen Our toy model. Bold: the high scores in each column. The nicknames describe what we observe; nobody built them in. The values were chosen by hand to be readable; in a real model they are learned.

Looking up

Turning a token into a vector is just a lookup: take the ID, go to that row, copy its three numbers. "speak" becomes [0.3, 0.9, 0.0]. There is no calculation and no intelligence involved. A real model also adds a few numbers that encode each token's position in the sentence, so it knows the order of the words; we leave them out to keep the toy simple.

Where the numbers come from

Before training, every box of this table holds a random number. Training then adjusts them with the same loop as the spam neuron: guess, measure the error, adjust. The third article explains where the answers come from for a language model. What matters here is the effect of those adjustments on the table.

Training never learns what a language is. What it discovers is that "French", "English" and "German" can replace one another in millions of sentences: after "I speak", before "fluently", next to "translate into". Every correction that helps the model predict what follows "French" also helps it with "English", so the two rows get pushed in the same direction, again and again. In the end, they hold similar numbers. Tokens that keep the same company end up with similar rows.

The table before and after trainingBefore training, tokens sit at random positions; after training, tokens used in similar contexts have drifted together.Before trainingRandom numbersAfter trainingSimilar company, similar placeFrenchEnglishtruthloudlyspeakIyouFrenchEnglishtruthloudlyspeakIyoubillions ofsmall nudgesHumans choose the size of the table. Training chooses every number in it.
learned Illustrative. In practice the starting values are random but small, so every token starts bunched near the centre of the map.

Naming the columns, after the fact

Now look at the finished table and ask a simple question: which tokens score high in each column? In Column 1, they are "Esper", "French", "English" and "fluently": tokens to do with languages. In Column 2, "speak", "truth", "fluently" and "loudly": tokens to do with speaking. In Column 3, "I" and "you": persons. So we give the columns nicknames: "language", "speaking", "person".

The order matters, and the nicknames come next. They describe what we observe in the numbers; nobody put them there. From here on we will use them, because "the language column" reads better than "Column 1", but they remain labels stuck on afterwards.

One caveat. In our toy, each column lines up neatly with one idea, because we chose the numbers to make the example readable. In a real model, with thousands of columns, it is rarely that tidy: an idea such as "language" is usually spread across many columns at once, and a single column mixes several ideas. Researchers can still find directions in a trained model that match ideas a human would recognise, and studying them has become a field of its own, called interpretability[3]. In these models, the meaning is never designed in advance: it can only be found after training.

What the numbers say

Three numbers per token make each token a point in a three-dimensional space. To draw it on a flat screen or page, we keep only two of the three columns, "language" and "speaking", and set the third aside:

Toy vocabulary on a mapTokens placed by their language score (horizontal) and speaking score (vertical).Column 1, nicknamed "language" →Column 2, nicknamed "speaking" ↑011LanguagesSayingFrenchEnglishtruthloudlyspeakfluentlyEsperantoIthe
learned Only two of the three columns are drawn, so the map fits on a page. "I" and "you" overlap here because they differ only in Column 3, which is not shown.

Tokens used in similar ways land close together: "French" and "English" are neighbours, and so are "truth" and "loudly". And "speak" sits on its own, between the two groups. Its row is the same whether the sentence ends in "French" or in "the truth": the table alone cannot know which meaning is meant. "Esper" and "anto", the two pieces of a rare word, are weak on both columns. Fixing all this is the job of attention, in the next section.

Observed, not guaranteed

Nothing in this method guarantees that useful meanings will appear. Researchers set up the conditions (a large table, a lot of text, a rule for adjusting the numbers), let training run, and then examined what had come out. The first time this was done on a large scale, in 2013, the result was striking: words had organised themselves by meaning, without anyone telling them how[4]. The same approach has been repeated at every new scale since, and the result has held each time. Even the "scaling laws", which predict how much better a model gets with more data and computing power, are regularities found by experiment, not mathematical theorems[5].

As with the spam neuron (first article, chapter 3), this is something observed and measured after the fact, not something that can be proven in advance.