03Cleaning it up

Why raw web text is mostly unusable, and the curation steps that make it fit for training.

Raw web text is mostly unusable: menus, cookie banners, spam, duplicated pages, machine-generated filler, personal data. Most of the engineering effort goes into curation. Meta describes its pipeline for Llama 3[2], and the other labs follow broadly similar steps:

From raw web pages to training dataRaw pages, extract the text, remove unsafe and personal data, remove duplicates, keep quality text, mix the sources.Raw pagesWeb crawls, code, books, papers, licensed dataExtract the textStrip menus, ads, page layoutRemove unsafe and personalAdult sites, personal-data-heavy domainsRemove duplicatesSame page, near-copies, repeated linesKeep quality textRules, then models that grade textMix the sourcesDecide the proportionsTokens, ready to train
The main curation steps, as described by Meta for Llama 3. Other labs follow broadly similar pipelines.
  • Extract the text. Keep the content of each page and throw away the layout, menus and advertising around it.
  • Remove unsafe and personal content. Meta removed domains known for adult content and domains holding large amounts of personal information.
  • Remove duplicates. The same article copied across a thousand sites would otherwise be learned a thousand times. Duplicates are removed at the level of addresses, whole documents, and even individual lines repeated across millions of pages.
  • Keep quality text. Simple rules first: too many repeated phrases, too many offensive words, a vocabulary that looks machine-generated. Then models trained to grade text. Meta used its previous model, Llama 2, to label examples of good and bad text, and trained the grading models on those labels. Each generation of model helps select the reading list of the next.
  • Mix the sources. The proportions are a deliberate choice, tested on smaller models first. Llama 3's final mix was roughly half general knowledge, a quarter mathematics and reasoning, 17% code and 8% other languages. That last figure is one reason these models are noticeably stronger in English than in French.

Two remarks for executives. First, a model can only be as good, as balanced and as current as its reading list. Its knowledge stops at the date the text was collected, and its gaps and biases are largely those of its data. Second, the legal status of training on copyrighted text is being contested in court in several countries. In the European Union, the AI Act now requires providers of general-purpose models to publish a summary of the content used for training and to respect rights holders' opt-outs[4].