On 12 June 2017, eight authors at Google published Attention Is All You Need. It is now the most consequential machine learning paper of the century and every frontier model in service today is a descendant of it.
It also got its own title wrong, in a way that is genuinely instructive.
Attention was not new in 2017. It had been bolted onto recurrent translation models for three years. What the paper removed was recurrence, and the reason that mattered has almost nothing to do with attention being a better way to relate words. It has to do with what a GPU can do while it waits.
Nine years is long enough to run a proper audit. This paper is that: what the 2017 architecture actually proposed, which of its decisions survived contact with scale, which were quietly replaced, and why the one flaw nobody has solved still turns up on your invoice. We sketched the longer lineage in Brain blueprints of AI; this goes into the single paper in detail.
1. What the paper actually proposed
Worth being concrete, because the specifics are what the audit is about.
The Transformer was an encoder decoder model for translation. Six encoder layers, six decoder layers, a model width of 512, eight attention heads of 64 dimensions each, and a feed forward inner dimension of 2048. The base model was around 65 million parameters, the large one around 213 million.
The attention mechanism is one line: take queries, keys and values, score every query against every key with a dot product, divide by the square root of the key dimension, softmax, and use the result to weight the values. The division is not decoration. Without it, dot products grow with dimension, the softmax saturates, and gradients vanish.
Because attention treats its input as a set rather than a sequence, position had to be injected separately, which the paper did with fixed sinusoids of varying frequency. Normalisation went after each sublayer, the feed forward network used ReLU, and every sublayer was wrapped in a residual connection.
The results: 28.4 BLEU on WMT 2014 English to German and 41.8 on English to French, the latter after three and a half days on eight GPUs. The abstract is careful to note that this was a small fraction of the training cost of the models it beat.
That sentence is the whole story, and almost nobody quotes it.
2. The contribution was not attention
Here is the argument. Consider what it costs to mix information across a sequence.
A recurrent layer processes position by position: to compute step n you need step n minus one. The number of sequential operations is proportional to sequence length. Put that on a GPU with thousands of cores and most of them wait, because the dependency is in the algorithm and no amount of hardware removes it.
Self attention has no such dependency. Every position attends to every other position simultaneously, so the number of sequential operations is constant, regardless of length. The paper's own complexity table makes this the headline comparison, and it is the row that mattered.
So the Transformer's real contribution was hardware alignment. It replaced an algorithm that forced a GPU to idle with one that could saturate it. Everything downstream, the scaling laws, the willingness to spend tens of millions on a training run, the entire economics of the field, follows from the fact that throwing more parallel silicon at this architecture actually helps.
Attention was the mechanism. Parallelism was the payload. If a different set based mixing operation had been equally expressive, it would very likely have worked too, and some of the alternatives proposed since suggest exactly that.
This is a recurring pattern in our own work: the constraint that decides an outcome is rarely the one in the announcement. We made the same argument about model launches in AGI talk measures the wrong axis, and about compute economics in the GPU query engine paper.
3. What survived
Scale was the experiment nobody had run in 2017, and it sorted the paper's decisions into two piles. The structural ones held.
The block shape. Self attention, then a feed forward network, each wrapped in a residual connection and a normalisation step. Stack the blocks. That skeleton is essentially unchanged in every frontier model today.
Multiple heads. Letting different heads attend to different relationships turned out to be load bearing rather than a nicety.
The scaled dot product. Still the operation, still divided by the square root of the key dimension.
Residual connections. Inherited from ResNets rather than invented here, but without them nothing deep trains.
That is a short list. Now the other pile.
4. What was quietly replaced
Almost every specific parameterisation in the paper has been swapped out, each for a documented reason.
| 2017 | Now | Why |
|---|---|---|
| Encoder decoder | Decoder only | The original was a translation model. Generative pretraining needs one stack |
| Sinusoidal positions | Rotary embeddings | Encodes relative position by rotating query and key vectors, and extrapolates better |
| Post normalisation | Pre normalisation | Normalising before the sublayer stabilises gradients deep enough to train at all |
| LayerNorm | RMSNorm | Drops mean centring for a small compute saving at no quality cost |
| ReLU feed forward | SwiGLU and gated variants | Gated linear units measurably outperform the plain version |
| Dense feed forward | Mixture of experts | Capacity without proportional compute, by routing to a few experts per token |
| Multi head attention | Grouped query attention | Sharing key and value heads shrinks the cache that dominates inference memory |
Two of those deserve a note.
Pre normalisation is why depth works. The 2017 paper put normalisation after each sublayer, which is fine at six layers and becomes a training stability problem at sixty. Moving it before the sublayer, analysed properly in 2020, is the unglamorous change that made very deep stacks trainable without elaborate warmup schedules.
Grouped query attention is an inference decision, not a quality one. It exists because generation caches keys and values for every previous token, and that cache, not the weights, becomes the memory bottleneck. That is the same economics we costed out in the coding model comparison, where cache read pricing decided the bill.
And FlashAttention belongs in its own category, because it changed nothing about the mathematics. It reorganised the memory access pattern after observing that attention was bound by moving data between levels of GPU memory rather than by arithmetic. Identical outputs, dramatically less memory traffic. An algorithm nobody changed became practical at long context because somebody read the hardware manual.
5. Attention is not, in fact, all you need
The architecture contradicts its own title, and the evidence is in the parameter count.
In a standard block, the feed forward network holds roughly two thirds of the parameters. Attention is the part everyone teaches and draws; the feed forward layers are where most of the capacity sits.
What are they doing? In 2020, Geva and colleagues argued that feed forward layers operate as key value memories: the first matrix matches input patterns, the second retrieves an associated distribution over the output. On that reading, attention routes information between positions, and the feed forward layers store what the model knows.
The strongest evidence is what labs did when they wanted more capacity without more compute. They did not scale attention. They scaled the feed forward path, splitting it into experts and routing each token to a few of them. That is the design in Kimi K3 at 2.8 trillion parameters, which we costed in the token economics note, and in Inkling at 975 billion with roughly 41 billion active, from the fine tuning paper.
Both are enormous because the part the title dismissed turned out to be where knowledge lives.
6. The flaw nobody solved
Self attention compares every position with every other position. Cost grows with the square of sequence length. That was irrelevant for translating sentences and it is the central constraint of everything since.
Nine years of attempts: sparse patterns, low rank approximations, linear attention, recurrence smuggled back in through state space models, and the sliding window and hierarchical schemes in current production models. FlashAttention pushed the constant down so hard it bought several years, and ALiBi and rotary embeddings improved extrapolation beyond the trained length. None of it changed the exponent.
You can see the bill. GPT-6 Astra reprices requests above 272,000 input tokens at twice the input and cache rates, applied to the whole request. Long context is a premium product because the underlying operation is quadratic, and every long context price list is a restatement of a 2017 design decision.
This also explains why the industry cares so much about caching. If recomputing attention over a long prefix is expensive, not recomputing it is the highest leverage optimisation available, which is why cache read pricing now decides which model is cheaper for agentic work.
7. What we take from it
Three things, and the third is why we wrote this.
The durable idea was hardware alignment. Not attention, not the specific block, but a design whose sequential depth is constant so parallel hardware can be saturated. Any architecture that wants to displace it has to beat it on that axis first, which is why several elegant alternatives have not.
The title was a slogan, and slogans age badly. Attention was not all you need, as the parameter distribution and the entire mixture of experts literature demonstrate. The paper is not diminished by this. It is a useful reminder that the compressed claim and the technical contribution are different objects, and only one of them travels.
Almost everything specific in a landmark paper gets replaced. The encoder decoder, the positions, the normalisation, the activation, the attention variant. What survived was the shape of the thing, not its settings. That is worth remembering when reading any current architecture announcement: ask which part is structural and which part is a parameterisation somebody will swap out within eighteen months.
The transformer's real lesson is not that attention was magic. It is that a design which fits the machine it runs on will beat a more elegant one that does not, and that the industry took nine years to exhaust the implications of three and a half days on eight GPUs.
If you are working out which architectural decisions matter for a system you are actually building, talk to us, or see how we build systems for businesses.
8. Sources
- Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, Attention Is All You Need, arXiv 1706.03762, June 2017. All figures for the original architecture are from this paper.
- Su et al, RoFormer: enhanced transformer with rotary position embedding, arXiv 2104.09864.
- Xiong et al, On layer normalization in the transformer architecture, arXiv 2002.04745, on pre normalisation.
- Zhang and Sennrich, Root mean square layer normalization, arXiv 1910.07467.
- Shazeer, GLU variants improve transformer, arXiv 2002.05202, and Fast transformer decoding: one write head is all you need, arXiv 1911.02150, on multi query attention.
- Ainslie et al, GQA: training generalized multi query transformer models from multi head checkpoints, arXiv 2305.13245.
- Dao et al, FlashAttention: fast and memory efficient exact attention with IO awareness, arXiv 2205.14135.
- Geva et al, Transformer feed forward layers are key value memories, arXiv 2012.14913.
- Press, Smith and Lewis, Train short, test long: attention with linear biases enables input length extrapolation, arXiv 2108.12409.
Architecture details for current commercial models are not fully published, so the statements here about what frontier systems use are drawn from the open literature and from open weight models whose designs are documented. The parameter split between attention and feed forward varies with width and expansion ratio; the two thirds figure describes a standard block at the usual four times expansion, not every configuration.
This note sits in our Research track, alongside the foundation models and GPU acceleration threads. The full library is at Research.