Attention
The mechanism that changed everything
How attention made it possible to look directly at what’s relevant and opened the door to Transformers.
So far we’ve traveled a path: we saw how RNNs tried to remember, how LSTMs and GRUs learned to control memory with gates, and how Beam Search helped generate text sensibly.
But there was a glass ceiling that no recurrent architecture managed to break:
Processing sequences step by step is too slow and too rigid.
Attention was born precisely to tear down that limit.
The problem attention came to solve
The classic architectures (RNN, LSTM, GRU) dragged along two very heavy chains:
1. They were strictly sequential. Each step depended mandatorily on the previous one. It’s like an assembly line where the second worker can’t start until the first finishes: impossible to take advantage of the brute power of modern GPUs, which are designed to do thousands of things at once.
2. Memory was still a bottleneck. Even though LSTMs and GRUs improved retention, they were still forced to compress all the meaning of a text into a single hidden state vector.
The sticky-note analogy. Imagine reading a 500-page novel word by word and, when you finish, trying to summarize every exact detail on a single sticky note. If someone then asks you about a detail from page 12, it’s almost certain it will have already been erased or blurred.
Attention proposed a radical shift in mindset:
«What if, instead of forcing the network to remember everything at once on a sticky note, we let it keep the book open and look directly at the key pages when it needs to answer?»
The core idea: Query, Key, and Value
To understand attention without getting lost in the algebra, think of a search engine like YouTube or a library:
- Query ($Q$): what you’re searching for right now (the word or concept you want to decode).
- Key ($K$): the title or label of each available item in the sentence.
- Value ($V$): the actual information each word contributes.
The mechanism follows a three-step logic:
- It compares the Query against all the Keys to measure how relevant each word in the sentence is.
- It converts those relevance scores into importance percentages (weights that add up to 100%).
- It extracts and combines the Values, giving more weight to the most relevant words.
Mathematically: the formula that moves the world
In the foundational paper it was defined through Scaled Dot-Product Attention:
Read in plain language, it’s astonishingly intuitive:
- $Q K^T$ (dot product): measures the affinity between what I’m looking for ($Q$) and what’s available ($K$). If they match, the result is high.
- $\sqrt{d_k}$ (scaling): a simple correction factor to prevent the numbers from growing too large and destabilizing the gradients (remember the exploding gradient from the previous chapter).
- $\text{softmax}(\dots)$: turns those affinity scores into probabilities (weights between 0 and 1).
- $V$ (values): multiplies those weights by the actual content of each word.
It is, literally, a mechanism that computes a dynamic highlighter over the text: it underlines what matters more strongly and leaves the irrelevant in gray.
Self-Attention: words talking to each other
In modern language models, the reigning variant is Self-Attention.
Instead of comparing two different sentences (as in English → Spanish translation), each word in a sentence is compared against all the other words in its own sentence.
Look at this classic dilemma:
«The animal didn’t cross the street because it was too tired.»
Who was tired? The animal or the street? Through Self-Attention, the word «it» computes its affinity with all the others and discovers that its greatest attention weight points straight to «animal». It doesn’t need pre-programmed rules: the context emerges from pure interaction between the words.
This is what makes it possible to capture global context, resolve ambiguities, and model meaning in a distributed way. Self-attention is, without exaggeration, the heart of the Transformers.
Multi-Head Attention: looking from several angles
Humans don’t process a sentence by paying attention to a single thing at a time. When you read:
«The central bank raised rates because it feared inflation.»
Your brain simultaneously connects syntactic relationships (who performs the action), semantic ones (what kind of bank it is, not the one you sit on), and cause-effect ones (why it did it).
Transformers imitate this with Multi-Head Attention: instead of a single focus, they split the computation into several parallel «heads,» each specialized.
One head watches grammatical agreement, another connects distant pronouns, another detects tone. Then all those views are merged into an enriched representation. It’s like having several expert readers analyzing the same sentence at once, each with their own specialty.
The paradigm shift: Recurrence vs. Attention
Attention proved to be superior in every critical aspect of language processing:
| Challenge | RNN / LSTM / GRU | Attention |
|---|---|---|
| Connecting distant words | Hard (the signal dilutes along the way) | Direct (compute distance = 1) |
| GPU parallelization | Impossible (step-by-step mandatory) | Total (the whole matrix at once) |
| Training speed | Slow | Extremely fast |
| Resulting context | Compressed into a single vector | Dynamic and global for each token |
| Scalability | Limited by the sequence | Massive (billions of parameters) |
Attention isn’t an incremental improvement: it’s a paradigm shift. We went from reading word by word to reading the whole sentence at once and highlighting what matters.
«Attention Is All You Need» and the birth of Transformers
Until 2017, attention was used only as a «support accessory» to improve recurrent networks.
But that year, researchers at Google published the historic paper: «Attention Is All You Need.» Its thesis was devastating:
«We can throw out RNNs, LSTMs, and convolutions. Attention, on its own, is enough.»
That’s how the Transformer architecture was born, built solely on:
- self-attention
- multi-head attention
- feed-forward layers
- normalization (LayerNorm)
- residual connections
No recurrence. No convolutions. Just attention.
That design eliminated the sequential bottleneck and made it possible to scale training over massive volumes of internet data, directly giving rise to GPT-4, Claude, Gemini, and the entire current generative AI revolution.
In summary
Attention redefined sequence processing:
- It removed the bottleneck of compressing all information into a finite memory.
- It enabled total parallelization, allowing training in times previously unfeasible.
- It created dynamic context, where each word is redefined according to those around it.
It’s the bridge between recurrent architectures and Transformers. It was the exact moment when language processing stopped reading «blindly and in a hurry» and began to analyze language in a truly contextual way.