05See for yourself how it works

The whole toy model in one animation, from "I speak Esperanto" to the next word, step by step.

Everything described in this article, in one place, with the numbers of our toy model. Press Play, or step through at your own pace, and watch "I speak Esperanto" become tokens, IDs and vectors, then watch "anto" gather context through attention, get corrected by the classic network, and produce a prediction. Every number on screen is one you can find, and check, in chapters 1 to 3, and replay in the companion spreadsheet.

Animated walkthroughStep by step, I speak Esperanto becomes tokens, IDs and vectors; anto gathers context through attention, is corrected by the classic network, and the model predicts fluently.I speak EsperantoIID 0[0.0, 0.1, 0.9]␣speakID 1[0.3, 0.9, 0.0]␣EsperID 2[0.6, 0.2, 0.0]antoID 3[0.5, 0.1, 0.0]2%22%49%27% itselfcorrection [−0.4, +0.4, 0.0]Probability of coming next␣fluently76%␣speak15%␣English5%␣French2%␣loudly1%␣fluentlyappended