06Watching it learn
What engineers watch while a model trains, and why it improves on texts it has never seen.
Training is not launched and left alone. The engineers watch:
- The error, on text the model has never seen. A slice of data is set aside and never used for training. If the error goes down on that unseen text too, the model is learning patterns rather than memorising.
- Checkpoints. The full state of the model is saved regularly, so training can resume after a failure or roll back if something goes wrong.
- Failures, which are routine. Over a 54-day period, Meta's training run, spread over up to 16,000 GPUs, was interrupted 466 times, 419 of them unexpectedly, mostly because of hardware faults[2]. At that scale, keeping the machine running is a discipline of its own.
- Step size. It starts small, rises, then decreases gradually towards the end of training, following a schedule chosen in advance and validated on smaller models. Like the data mix, it is a choice made by experiment, not derived from theory.
Why it works on texts it has never seen
Why does the model get better at texts it has never read? Because it has far too few parameters to memorise everything it is trained on, so the only way to keep improving is to capture what all those texts have in common: grammar, facts, patterns of reasoning. Like a student facing millions of different exam questions, it has no choice but to understand rather than cram. Patterns, unlike memorised answers, carry over to new questions.
Measured, not proven
This also means there is no proof that its answers are correct. As the first article explained (chapter 3, "Proven, or measured?"), the whole approach rests on an observation, not a proof: that useful patterns emerge from enough text and enough adjustments. It has held so far, but a model's reliability is measured by testing, not demonstrated in advance, much as we trust a pilot by their record rather than by a theorem. That is why these systems are mostly right, never guaranteed right, and why they need verification around them. Researchers themselves are still working out why this works as well as it does: in this field, theory lags behind practice.