02Neurons and layers
What an artificial neuron calculates, how neurons form layers, and why a layer is just a table of numbers.
The transformer is a neural network, so before going further, it helps to know what a neuron does. In AI, the word is borrowed from biology, and the borrowing is loose. An artificial neuron is a very small calculation, simple enough to do on the back of an envelope.
One neuron: a weighted vote
Take a task every company knows: deciding whether an email is spam. A neuron looks at a few clues, each with a value of 1 (yes) or 0 (no). Each clue is a check that a few lines of code can do: search for a word, or look up the sender in the address book.
- Does the email contain the word "free"?
- Is the sender unknown?
- Does it mention my name?
Each clue has a weight, a number that says how much it counts and in which direction. "free" counts strongly towards spam (+2). An unknown sender counts a little (+1). My name in the email counts against spam (−2), since spammers rarely know it. The neuron multiplies each clue by its weight, adds everything up, and compares the total with a threshold, here 2.5. If the total exceeds the threshold, the neuron "fires" and passes on how far above the threshold it is. Otherwise, it stays silent and passes on 0.
That is all a neuron does: multiply, add, compare with a threshold, pass on the result or nothing. The intelligence, if any, is in the weights. With the wrong weights, the same neuron would flag your colleagues and let the spam through. Choosing good weights is called training, and chapter 3 shows how it works.
The idea is old, as chapter 1 recalled, and the brain inspired it only loosely: a biological neuron also receives signals, weighs them and fires above a threshold, but the resemblance stops there.
Inside a neuron: two steps
Step one: a weighted total
The neuron multiplies each input by its weight and adds everything up. It then adds one more number of its own, called the bias, which plays the role of the threshold: a threshold of 2.5 is the same as a bias of −2.5. For the spam email of our example: 1 × 2 + 1 × 1 + 0 × (−2) − 2.5 = 0.5. This first value can be any number, large or small, positive or negative. On its own, it means little.
Step two: a bend
The total then passes through a fixed function, the same for every neuron of a layer, called the activation function. Three are worth knowing:
Why does this bend matter? For three reasons.
- It gives the output a meaning. The sigmoid turns any total into a score between 0 and 1 that reads like a degree of confidence: 0.86 means "fairly sure it is spam", 0.27 means "probably not". The decision is then a matter of where you set the line, usually at 0.5.
- It allows learning. With the original yes-or-no step[6], a small change in the weights usually changes nothing in the output, so training cannot tell whether it is getting closer. With a smooth curve, every small change in a weight produces a small change in the score, and that is what tells training which way to adjust.
- It makes layers worthwhile. Without a bend, a layer is just weighted totals, and weighted totals of weighted totals add up to a single weighted total. However many layers you stacked, they would do no more than one. The bend is what lets a second layer build "patterns of patterns", as the next example shows. Large networks today mostly use the simplest bend of all, ReLU, or smoother variants of it[7].
Many neurons: layers
A single neuron can only weigh the clues it is given. It cannot express "suspicious when two things happen together". Take a harder question: is this email a phishing attempt? No single clue settles it. An urgent tone is normal from a colleague; a link is normal in a newsletter. What gives phishing away is a combination.
The solution is to stack neurons in layers. The neurons of a first layer each look at the raw clues and detect a simple pattern. The neurons of the next layer do not look at the email at all. They look at the outputs of the first layer and combine them:
The negative weight matters: a weight can count against a conclusion. Each layer builds on the one below it: patterns, then patterns of patterns. Image recognition gives the most famous example. In networks trained on photographs, researchers found early layers that detect edges, middle layers that combine edges into shapes, and later layers that combine shapes into faces or objects[8]. Nobody programmed those stages. They appeared because they helped the network do its job.
Where do the clues come from?
In our two examples, the clues are prepared by a simple program: it searches the email for a word, or looks up the sender in the address book. A human chose which clues to check. For decades, this was how machine learning worked. Engineers spent most of their effort designing the right clues for each problem, a craft called feature engineering, and the network only learned how to weigh them. If a useful clue was missing from the list, no amount of training could make up for it.
The major change of the last fifteen years is that networks now learn their own clues. Give a network raw material and enough layers, and its first layers learn to extract useful clues, which the next layers combine. Nobody tells it what to look for. The turning point is usually dated to 2012, when a deep network trained on raw images won a major image-recognition competition by a wide margin[9].
A transformer works this way. It receives no prepared clues about the text, only the text itself, cut into pieces by a mechanical procedure. The second article explains it, as the first thing a language model must do differently from our spam filter. Everything after that, including what to notice and what to combine, is learned during training.
A layer is a table
There is a simpler way to look at a layer, and it will matter for the rest of this series. Take the phishing network. Write the weights of each neuron of layer 1 on one row, with one column per clue and the bias at the end. Do the same for layer 2, which has a single neuron, "Phishing", reading the two outputs of layer 1. The whole network is now two small tables. Let's run two emails through it. They are identical (urgent wording, a password request, a link), except that email A comes from an unknown sender and email B from a contact.
Follow email A. In layer 1, each row is multiplied by the email's clues and added up: "Pressure" totals 2 + 2 − 3 = 1, "Dubious link" totals 2 − 1 = 1. The bend of layer 1 lets positive totals through unchanged, so both pass on 1. In layer 2, "Phishing" multiplies those two outputs by its own weights: 1.5 + 1.5 − 2 = 1. Its bend, a sigmoid, turns that total into a score of 0.73, above 0.5: phishing.
Now email B. The only difference is the contact, which carries a weight of −3 in the "Dubious link" row. Its total drops to 2 − 3 − 1 = −2, and the bend silences it to 0: since the sender is known, the link raises no suspicion. Layer 2 now receives 1 and 0, and totals 1.5 − 2 = −0.5, a score of 0.38: not phishing. One clue, through two tables and two bends, changed the verdict.
That is the complete network: clues, table, bend, table, bend, verdict. A larger network only has more columns, more rows and more tables.
Tables, matrices and vectors
These tables are not just a convenient picture. In mathematics, a table of numbers arranged in rows and columns is an object in its own right, called a matrix, and a single row or column of numbers is called a vector. They belong to a well-established branch of mathematics, linear algebra, which has studied for more than a century what can be done with them and how to do it efficiently.
Everything a neural network does is computation on matrices. The weights of a layer form a matrix. The inputs it receives form a vector. Computing the layer, row by row, multiply and add, is what mathematicians call multiplying a matrix by a vector. Process a thousand emails at once, and the thousand input vectors stack into a second matrix: the whole batch becomes a single multiplication of one matrix by another. When the next article "slices and dices" text, it is always this: a very large number of very simple calculations, organised as operations on matrices.
This is also why the vocabulary of AI is full of these words. The rows of numbers that represent tokens are called vectors (second article, chapter 1). The "match" between two of them, multiply column by column and add, is what mathematicians call a dot product (second article, chapter 2). And when a matrix has more than two dimensions, a stack of tables rather than a single one, it is called a tensor, which is where the "tensor cores" of graphics processors get their name.
Three consequences follow:
- A network is a stack of tables. Each layer is one table of weights, applied one after the other, with a bend in between. Every number in every table is a weight, and the weights are what training adjusts.
- The tables get very large. Our tables have a handful of cells. A layer that reads 4,000 numbers and holds 16,000 neurons is a table of 64 million weights, and a large language model stacks dozens of such layers. When people say a model has "billions of parameters", these are the numbers they are counting.
- It is the same operation everywhere. Multiply and add, repeated over huge tables, and repeated again for every email, or every word, processed. Hardware that can do thousands of these multiplications at the same time, which is exactly what graphics processors (GPUs) do, therefore has a decisive advantage. The third article in this series explains why this makes GPUs so central, and so expensive.