02Where the text comes from

Where the training text comes from, and how much of it a frontier model reads.

A model learns only from what it reads, so the first job is to gather an enormous amount of text. The main sources:

  • The public web. Most of the volume. Much of it comes from Common Crawl[1], a non-profit that has been archiving the public web since 2008, supplemented by the labs' own crawlers.
  • Code, from public repositories.
  • Reference and scientific text: encyclopedias, academic papers, patents, technical documentation.
  • Books and licensed data, bought or negotiated from publishers, news organisations and data providers.
  • Synthetic data, increasingly: text produced by other models, for example worked mathematics problems, checked before use.

The quantities are hard to picture. Meta's largest Llama 3 model was trained on about 15 trillion tokens[2]. A person reading eight hours a day for seventy years gets through roughly 3 billion words, so this amounts, very roughly, to several thousand lifetimes of full-time reading. Public datasets of the same order exist: Hugging Face's FineWeb holds 15 trillion tokens built from 96 Common Crawl snapshots, with every processing step documented[3].