07From text completer to assistant

The later phases of training that turn a model that continues text into an assistant that answers.

A model trained only on raw text continues text: ask it a question and it may well reply with more questions, as if continuing a list of exam topics. Later phases of training turn it into an assistant, and here the direction of each nudge comes from new sources:

  • Supervised fine-tuning. People write examples of good answers to questions, and the model is trained on them exactly as above: the answer key is the human-written reply.
  • Learning from preferences. For many questions there is no single correct answer, only better and worse ones. The model produces several answers, people rank them, and a second model learns to predict those rankings as a score. Training then nudges the model so the preferred answer becomes more likely and the rejected one less likely[7]. The machinery is the same; only the source of the direction changes: a score instead of a known next token. This is reinforcement learning from human feedback.
  • Learning from verifiable results. For mathematics or code, an answer can be checked automatically: the result is right, the program passes its tests. Recent "reasoning" models are trained largely this way: the model tries, the check scores the attempt, and successful lines of reasoning are reinforced[8].

The three phases map onto the three ways to learn described in the first article:

PhaseWhere the right answers come fromKind of learning
PretrainingThe text itself: the next tokenSelf-supervised
Fine-tuningAnswers written by peopleSupervised
Preferences and checksA score: human rankings, or an automatic testReinforcement learning

In every case, the direction of each nudge comes from the same question: what change would have made the desired output a little more likely? This phase is also where vendors shape tone, helpfulness, refusals and safety behaviour, which explains why models built on the same architecture can behave so differently.