08Why GPUs
Why language models run on GPUs, the two bottlenecks that limit them, and why the chips are so expensive.
Every step of a language model, as the first two articles showed, comes down to one operation: multiply each weight by an input and add up the results. A layer is a table of weights; computing it means doing that for every row. There is nothing else, no hidden complexity, just this operation repeated on an enormous scale. A useful rule of thumb: producing one token takes about two operations per parameter. For a model with 70 billion parameters, that is 140 billion operations, for every token of every answer.
Why this work suits a GPU
These operations have two properties that decide which hardware suits them.
They are all the same
Every cell of every table calls for the same thing: multiply a weight by an input, add the result to a running total. There are no decisions to make along the way and no exceptions to handle.
Most of them do not depend on each other
Go back to the phishing table in the first article. To compute the "Pressure" row, you need the email's clues and that row's weights, nothing else. The same goes for the "Dubious link" row. Two people could each take a row and work at the same time, without ever waiting for one another. In a layer of 16,000 neurons, 16,000 people could work at the same time. The only waiting happens between layers: layer 2 needs the results of layer 1 before it can start.
Now compare this with the work a company's servers usually do. Processing a bank transfer means checking the balance, then debiting one account, then crediting the other, then recording the transaction. Each step depends on the one before, and at each step the program may branch: insufficient funds, a blocked account, a currency to convert. That work is varied, full of decisions, and sequential.
A CPU is built for that second kind of work. Its few dozen cores are each very fast, very versatile, and full of circuitry to handle decisions and to keep one step flowing quickly into the next. A GPU makes the opposite choice. It drops most of that circuitry and uses the space for thousands of simple cores (16,896 on an H100, plus 528 tensor cores, circuits designed to multiply small matrices in a single step[9]), which all apply the same instruction at the same moment, each to its own piece of data. On varied, step-by-step work, a GPU would perform poorly. On a table of billions of identical, independent multiplications, it does in one pass what a CPU's handful of cores must take turns to do.
A CPU is like a small team of senior experts: each one can handle any task, with complex logic and decisions at every step. A GPU is like a stadium full of people who can only do one thing, multiply and add, but all at the same time. For a spreadsheet of a few hundred cells, the experts win. For tables of billions of cells, the stadium wins by a wide margin.
Two bottlenecks, not one
The cores are only half the story. Before multiplying, the processor has to fetch the weights from memory, and a model's weights are enormous: a 70-billion-parameter model occupies about 70 to 140 gigabytes, depending on how precisely each number is stored. Which limit matters depends on the job.
- Training is limited by computing power. The same weights are reused across millions of examples processed together, so what matters is how many operations per second the hardware can do. Training Meta's largest Llama 3 model took roughly 40 million billion billion operations (4 × 1025, using the standard estimate of six operations per parameter per training token[10]). On a single H100[9] running at a realistic 40% of its peak, that would take roughly 3,000 years. Spread over 16,000 GPUs, it took a few months[2].
- Answering is limited by memory. To produce each token, the processor must read every weight of the model once. The speed of the answer is set less by the cores than by how fast the weights can be read from memory. This gives a simple ceiling: memory bandwidth divided by model size[11].
Two lessons from this chart. First, a small model on a good server processor[12] is perfectly usable: 38 tokens per second is faster than anyone reads. Second, the gap widens when serving many users at once. A GPU reads its weights once and uses them for dozens of simultaneous requests, which multiplies its throughput, while a CPU quickly runs out of computing power. A GPU's advantage is modest for one person and enormous for a thousand.
Parameters are physical
The second article counted parameters: 50 in our toy, 405 billion in Llama 3. GPT-2, in 2019, had 1.5 billion[5]; GPT-3, a year later, 175 billion[13]. For today's leading models, the figures in the press reach into the trillions. Three things are worth knowing about these numbers:
- Most are estimates. The leading vendors no longer publish parameter counts for their frontier models. Figures quoted in the press are usually informed guesses.
- Not all parameters work at once. Some recent models are built as a set of specialised sub-networks, called "experts", and only a few are used for each token. DeepSeek-V3, for example, has 671 billion parameters in total but uses about 37 billion for each token[14]. That cuts the computation per token, but every expert must still sit in memory.
- They take up room. Each parameter is a number that must sit in memory while the model runs. A trillion parameters take up roughly two terabytes, far more than any single chip holds, which is why serving these models requires clusters of GPUs connected together.
Why GPUs are so expensive
An H100 costs in the order of 25,000 to 30,000 dollars, against a few thousand for a top server processor. Several reasons add up:
- Specialised memory. The memory that gives a GPU its bandwidth (called HBM) is stacked in layers right next to the chip. It is far more expensive to make than ordinary memory, and in short supply.
- One dominant supplier. NVIDIA's lead does not come from the chips alone. Its software platform, CUDA, has been the standard for fifteen years, and almost all AI software is built on it, which makes switching costly for buyers.
- Demand far above supply. Every large lab and cloud provider is buying at the same time, and production capacity for the most advanced chips is limited.
- Power and cooling. An H100 draws up to 700 watts, and a cluster of thousands needs dedicated data centres.
What this means in practice
- Using a frontier model through a vendor's API means renting a slice of their GPUs, paid per token. You never buy the hardware.
- Running a small open model on your own premises (a few billion parameters, compressed) is feasible on ordinary servers, or even a laptop, for modest volumes. This is what tools such as llama.cpp and Ollama are for (chapter 10).
- Running a large model on your own premises for many users does require GPUs, bought or rented by the hour. Sizing that investment means estimating three things: the model size, the number of simultaneous users, and the acceptable answer speed.