Christophe MontagneInterim CTO · Advisory

What Claude, Gemini and ChatGPT actually are/2 min read

01Why a new architecture?

Why recurrent networks hit a wall, and how attention became the whole design.

Neural networks are not new. The first mathematical model of a neuron dates from 1943[1], and by the late 1980s researchers knew how to train networks of many such neurons to recognise patterns[2]. But these networks had a blind spot: they take in their input all at once, as a fixed block, with no notion of before and after. That works for an image. It fails for language, where meaning depends on order. "The dog bit the man" and "the man bit the dog" contain exactly the same words.

The first serious answer was the recurrent network. Instead of reading a sentence in one go, it reads it word by word, carrying a kind of running memory from one word to the next. Refined in 1997 with a variant called LSTM[3], recurrent networks became the standard tool for translation and speech recognition through the early 2010s.

They had two weaknesses. First, their memory faded: by the end of a long sentence, the beginning was largely lost. Second, and more decisive, reading word by word means the work cannot be split up. Word ten has to wait for word nine. Graphics processors, the hardware that was transforming the field, are built to do thousands of calculations at once, and recurrent networks could not take advantage of them.

Recurrent network versus transformerA recurrent network reads words one after another; a transformer connects every word to every other word at once.Recurrent network: one word at a timeThedogbitthemanEach word waits for the previous one; early words fadeTransformer: all words at once, in parallelThedogbitthemanEvery word connects directly to every other word
Recurrent networks read in sequence; transformers process all words in parallel. (In models that generate text, each word only looks back at earlier words; see chapter 03.)

In 2014, researchers added a mechanism called attention to recurrent networks[4]. When translating a word, the model could look back over the whole source sentence and focus on the parts that mattered most. Translation quality improved markedly. But the recurrence, and its slowness, remained.

The Google team behind the 2017 paper asked a bold question: what if attention was not an add-on but the whole thing? Their title gave the answer: "Attention Is All You Need"[5]. By dropping recurrence entirely, they got a model that understood context better and, crucially, could be trained in parallel. Their translation model beat the state of the art after a few days of training on eight GPUs. That second property, speed at scale, is what would later make it possible to train models on a large share of the written web.

The rest of this article follows that design step by step. Here is the journey ahead:

Transformer pipelineTokens, vectors, attention, layers, next word; the stage covered in this chapter is highlighted.TokensCut the textVectorsMap of meaningAttentionUse contextLayersAttn + networkNext wordPredict↻ the predicted token is appended to the text and the cycle repeats