Try the toy model

Download the spreadsheetExcel file, 18 KB. Opens in Excel, LibreOffice or Google Sheets. Built with the help of Claude, Anthropic's AI assistant.

What is in it

The same model as in the article: twelve tokens, three numbers per token (language, speaking, person). Four sheets follow the article step by step:

• Vocabulary. The table of vectors from chapter 02. Each row is a token, and its three numbers are its vector. The sharpness setting is here too.
• Attention. How "anto" weighs the tokens before it and blends their vectors (chapter 05, step 2). A second block does the same for "I speak the truth".
• Next token. The classic network's correction, then the score and probability of every token in the vocabulary (chapter 05, steps 3 and 4).
• Training step. One wrong guess and the nudge that corrects it: towards the right answer, away from what the model expected (chapter 07).

Reading the cells

Blue numbers on a yellow background are inputs: change them freely. Black cells are formulas and update by themselves. Green cells are values carried over from another sheet.

Two conventions, as in the article. A match is the two vectors multiplied column by column, then added. Matches become percentages through the softmax function: each one is multiplied by the sharpness setting, raised as an exponential, and divided by the total. Vectors are rounded to one decimal at the end of each stage, so the spreadsheet gives exactly the numbers you read in the article.

Things to try

1. Change the meaning of a word. In Vocabulary, raise the "speaking" value of "French", say from 0.2 to 0.8, then look at Next token. "French" overtakes "fluently" as the most likely next word. The model knows nothing but these numbers: change them and you change what it believes.
2. Make attention sharper or softer. In Vocabulary, cell H4, change the sharpness from 10 to 2. In Attention, the shares become almost even: "anto" listens to everyone a little. At 30, one token takes almost everything. Real models learn this balance during training.
3. Switch off what the network learned. In Next token, set the network correction to 0, 0, 0. The model now hesitates between "English", "fluently" and "French". Attention alone gathers context; the knowledge of what usually comes next sits in the classic network.
4. Teach it something else. In Training step, change the right answer to "French", or the step size to 0.1 or 2. A small step barely moves the vector, a large one overshoots. This is why real training takes billions of tiny steps.
5. Give "anto" different company. In Attention, replace "Esper" with another token from the list, "truth" for instance. The shares change, and the vector of "anto" lands somewhere else on the map. Same token, different neighbours, different meaning.

The values in the vocabulary were chosen by hand to be readable, and the network's correction is stated rather than computed. In a real model, both are learned during training, and the columns have no labels.