01From a spam filter to an LLM

What changes between training a spam filter and training a large language model, and what stays the same.

The first article in this series trained a spam neuron: five clues, sixty labelled emails, a few hundred rounds of the same loop. Run the examples through, measure the error, work out which way each weight should turn, take a small step, start again. Training Claude, Gemini or ChatGPT uses exactly that loop. Three things change, and each is the subject of part of this article.

  • The answer key. Our emails were labelled by a person. Nobody can label trillions of words. A language model learns from text whose answers are already in it: hide the next word, and the text tells you what it was. Chapters 2 to 4 explain where that text comes from, how it is prepared and how it is fed in.
  • The scale. Five weights become hundreds of billions; sixty emails become trillions of words; a few seconds of calculation become months on thousands of specialised processors. Chapters 8 and 9 explain why this requires GPUs, and what it costs.
  • The goal. A spam filter must sort emails. An assistant must answer questions helpfully and safely, and for that there is often no single right answer. Chapter 7 explains how training adapts when the answer key is a human judgment rather than a fact.

We know exactly how a model learns. What we cannot predict is what it will end up knowing.