04Feeding it in

How cleaned text becomes training material, and why every chunk holds thousands of exercises at once.

The cleaned text is tokenised, documents are placed end to end, and the whole stream is cut into chunks of fixed length. For Llama 3, chunks were 8,192 tokens long during most of training[2].

Here is the key point, and it connects back to the beginning of the article. Remember the rule from the second article: each token can only look at the tokens before it. That rule turns every chunk into thousands of exercises at once. Each position is a question ("what comes next?") whose answer is simply the next token in the chunk:

Every position in a chunk is a training questionEach prefix of the chunk is a question whose answer is the next token.I␣speak␣Esperanto␣fluently␣and␣FrenchWhat the model seesAnswer to guessI␣speakI speak␣EsperI speak EsperantoI speak Esperanto␣fluentlyI speak Esperanto fluently␣andI speak Esperanto fluently and␣FrenchOne chunk of 7 tokens, 6 questions, all answered and corrected in a single pass.A real chunk of 8,192 tokens gives 8,191 questions at once.
The fourth row is the exercise worked through by hand in the second article.

Notice what this means. No human has to label anything: the text supplies its own answer key. The first article described three ways to learn, and this one does not fit any of them neatly. Each question comes with its right answer, as in supervised learning, but nobody wrote those answers: they were already in the text, as in unsupervised learning. Specialists therefore call it self-supervised learning: supervised in how it learns, unsupervised in where the answers come from. It is the reason training can scale to a large share of the written web[5]. It is also where the 2017 design pays off: all the questions of a chunk are answered and corrected in a single pass, where a recurrent network would have had to work through them one after the other.

Chunks are then grouped into batches. For its largest model, Meta started with batches of 4 million tokens and grew them to 16 million[2]. Each batch produces one step of learning, and about a million such steps took the model through its 15 trillion tokens. Most of the text is read only once. The best sources may be read several times, and the last phase of training concentrates on the highest-quality material, much like a final revision before an exam.