05What happens when a sentence is fed into a model

One sentence, “I speak Esperanto”, worked through by hand with the toy model, up to the next word.

The previous chapters took the transformer apart piece by piece. Now let's put it back together and follow a single sentence, "I speak Esperanto", from the moment it is typed to the moment the model predicts what comes next. We will do it with the toy model from chapter 02, so that every number can be checked by hand.

Step 1: from text to vectors

The symbol ␣ marks the space, which travels with the word that follows it.

First, the sentence is cut into tokens. "I" and "speak" are common enough to be tokens of their own, but "Esperanto" is not, so it is cut into two pieces, "Esper" and "anto". Notice that the space travels with the word that follows it: what the model actually receives is "I", " speak", " Esper", "anto".

Each token is then replaced by its ID, and each ID selects its row in the table of chapter 02. Our four tokens become I [0.0, 0.1, 0.9], speak [0.3, 0.9, 0.0], Esper [0.6, 0.2, 0.0] and anto [0.5, 0.1, 0.0]. A real model would also blend in a position signal at this point.

Who decides where to cut?

Not the model. The cutting is done by a separate and much simpler tool, the tokenizer, built before the model is trained. It knows nothing about meaning; it works purely on frequency.

The most common method, byte-pair encoding[15][16], starts from a large sample of text split into single characters. It finds the pair of neighbours that appears most often, say "e" followed by "a", and fuses it into a new piece. Then it does it again, and again, thousands of times, until the vocabulary reaches its target size, typically between 30,000 and 200,000 pieces.

Byte-pair encoding: the most frequent neighbouring pieces are fused, again and again, until the vocabulary reaches its target size [15][16].

Frequent words like "speak" end up as a single token; rare words like "Esperanto" stay in pieces, because the merges that would complete them never rank high enough. Once built, the tokenizer is frozen: the same text is always cut the same way. That is also why the cuts ignore grammar. "Esper" and "anto" are not meaningful parts of the word, just statistically convenient ones.

This mechanical detail has very practical consequences:

Language affects cost. Vendors bill per token, and tokenizers built mostly from English text cut other languages into more pieces. The same content in French usually costs more tokens than in English, and less widespread languages cost more still.
It explains some odd failures. Ask a model how many "r"s there are in "strawberry" and it may get it wrong: it never sees the letters, only a few chunks of the word.
Nothing is ever unknown. Any word can be built from smaller pieces, down to single characters if needed, so the model can always handle a new brand name, a typo or a product code.

Step 2: attention, by hand

Our four vectors now enter the first layer. Let's follow the last one, "anto", which starts as a weak fragment: [0.5, 0.1, 0.0].

How much does each earlier token matter? Attention starts by measuring how well "anto" matches each token up to and including itself. The measure is simple: multiply the two vectors column by column, then add. For "anto" and "speak": 0.5 × 0.3 + 0.1 × 0.9 + 0.0 × 0.0 = 0.24. The more two vectors point in the same direction, the higher the score. The scores are then turned into shares that add up to 100%, in a way that strongly favours the highest ones.

Blend. "anto" then takes a blend of the vectors it looked at, each in proportion to its share:

Values rounded to two decimals. You can check every line with a calculator.

Three things to notice in these numbers:

"anto" looks mostly at "Esper" (49%). It is completing its own word. It also takes a good share from "speak" (22%), and almost nothing from "I" (2%), which has little to contribute here.
The blend is added, not substituted. The update [0.50, 0.33, 0.02] is added to the original vector, as if writing in the margin of the same page. "anto" keeps what it was and gains what it learned from the others.
The result has moved. At [1.0, 0.4, 0.0], "anto" is now strongly about a language, and somewhat about speaking. It started as a fragment near nothing in particular. It now sits right next to "French" and "English".

The same arithmetic in "I speak the truth" pulls "truth" in a completely different direction. There, "truth" puts 85% of its attention on "speak", and its vector climbs on the "speaking" column instead: something being said, not a language. Same table, same rule, different company, different result. That is how the ambiguity of "speak" gets resolved.

Step 3: the classic network adds what it knows

The vector then passes through the classic network. Attention gathered context from the sentence. The network brings in knowledge learned during training, about what usually follows what. Across millions of training sentences, it has seen that once a language has been named after "speak", what comes next is usually about how it is spoken, not another language. So it applies a correction that lowers "language" and raises "speaking": [−0.4, +0.4, 0.0].

In a real model, the network computes this correction from the vector, using its own learned parameters. In our toy, we simply state it. Either way, the vector moves again: [1.0, 0.4, 0.0] becomes [0.6, 0.8, 0.0]. Here is the whole journey of "anto" on the map:

Toy model, first two columns. In a real model this journey happens over dozens of layers.

Step 4: choosing the next token

The final vector of "anto", [0.6, 0.8, 0.0], is now compared with every row of the table, using the same match as before: multiply column by column, add. For "fluently": 0.6 × 0.7 + 0.8 × 0.8 + 0.0 × 0.0 = 1.06. Each score is then turned into a probability, again in a way that strongly favours the highest:

Toy model. The percentages come from running the twelve-token model; they can be reproduced in a spreadsheet.

Try it yourself. Every figure in this chapter is computed live, so you can check each one and change it. Download the toy model as a spreadsheet, built with the help of Claude.

"fluently" wins with 76%. Notice what made this work. "anto" ended up close to "fluently" only because of two things: what it gathered from its neighbours through attention, and what the classic network knew about what follows a spoken language. Neither the token "anto" nor the table alone could have produced that answer.

From toy to real

Everything you just computed by hand is what a real model does, with four differences of scale:

Thousands of columns instead of three, with no readable labels.
Query, key and value. In our toy, each vector was used as is to compute the matches. A real model first transforms each vector three different ways, using three learned tables, so that a token can look for one thing while advertising another (chapter 03).
Many heads, many layers. We did one pass of attention and one network. A real model runs dozens of attention heads side by side and repeats the pair dozens of times.
A learned correction, and sometimes a separate output table. Our network correction was stated by hand; in a real model it comes from millions of learned parameters. And while some models compare the final vector with the same table used at the input, as we did, others use a separate output table.

The arithmetic, though, is the same: match, share, blend, add, correct, compare. In a real model, the climb looks like this, with context accumulating from left to right at every layer:

The coloured cells are a metaphor: in reality all the numbers of a vector shift slightly at every layer. The classic network is drawn with a wider middle layer, as in real models, where it is typically about four times wider than the vector.