Generative AI · Part 3 of 3
How Claude, Gemini and ChatGPT Are MadeTraining, data, GPUs and cost
Summary
The first two articles in this series explained how a neural network learns and how a large language model works. This third one explains how such a model is made, and what it takes.
Training an LLM uses the same loop as the spam filter of the first article: run examples through, measure the error, adjust every weight a little. What changes is the answer key, the scale and the goal. The text comes mostly from the public web, and most of the effort goes into cleaning it. The text supplies its own answers: hide the next word and check the guess, which specialists call self-supervised learning. Every wrong guess tells each of the model's billions of numbers which way to move, and the toy model shows exactly how. Later phases turn a text completer into an assistant, using human judgments and automatic checks.
The article then answers two practical questions executives ask: why this requires GPUs rather than ordinary processors, and what it costs. It ends with the libraries that let organisations use these models without rebuilding them.
Previous: part 2, What Claude, Gemini and ChatGPT Actually Are