LSTM + GRU
The networks that truly learned to remember
How LSTMs and GRUs solved the memory problem in sequences and paved the way toward attention and Transformers.
In the previous chapter we saw how RNNs tried to remember… and how the bidirectional versions tried to understand.
But both had an enemy they couldn’t defeat:
Remembering over the long term is hard.
The vanishing gradient made RNNs forget important information right when they needed it most. LSTMs and GRUs were born to solve exactly that problem.
The problem that had to be solved
A basic RNN updates its memory like this:
Each step blends the new ($x_t$) with the old ($h_{t-1}$). The trouble is that, when the sequence is long, that repeated blending makes the old memory dilute until it vanishes.
One way to picture it. It’s like trying to remember a long sentence by repeating it in your head, but each time you repeat it a little more quietly… until nothing is left. The information from the beginning fades before it reaches the end.
LSTMs and GRUs proposed a different idea:
«Instead of blending everything blindly, let’s control what to remember, what to forget, and what to update.»
And that changed everything.
LSTM: Long Short-Term Memory
An LSTM is an RNN with gates: small mechanisms that decide what information comes in, what goes out, and what stays.
It has three main gates.
1. Forget gate. Decides which part of the previous memory should disappear.
2. Input gate. Decides what new information is worth adding.
3. Output gate. Decides which part of the memory is used to produce the output at this step.
Notice the $\sigma$ function (the sigmoid): it returns values between 0 and 1. It’s literally a regulator: 0 means «let nothing through,» 1 means «let everything through,» and the values in between let a portion pass. Each gate is like a faucet the network learns to open or close.
And on top of that, the LSTM keeps a cell state $c_t$: the «pure» memory, which travels along the entire sequence with very little distortion. That’s the key to its success.
The intuition behind LSTMs
Think about how your own memory works:
- there are things you forget on purpose
- things you decide to remember
- things you use in the moment
- things you store for later
An LSTM does exactly that, but with math. It’s like having a filing cabinet with three buttons: FORGET, STORE, and USE.
A concrete example. Let’s revisit the sentence that broke basic RNNs:
«I was born in France… (thirty words later) …that’s why I speak fluent ______.»
An LSTM can open the input gate when it reads «France,» store that fact in the cell state, keep the forget gate nearly closed during the thirty intervening words (which add nothing relevant), and finally use that memory when it reaches the blank to answer «French.» That’s why LSTMs remember information across hundreds of steps where an ordinary RNN would already be lost.
Why were they revolutionary?
Because they solved the problem RNNs couldn’t: handling long-range dependencies, complex context, long sentences, distant relationships, and doing it with stability during training.
It’s no exaggeration to say LSTMs were the standard in NLP for nearly a decade, from machine translation to speech recognition.
GRU: Gated Recurrent Unit
The GRU was born as a simpler version of the LSTM. The idea was: do we really need three gates and two separate memory states?
It has only two gates.
1. Update gate. Decides how much of the previous memory to keep versus how much new information to bring in. It merges into a single mechanism the work that the forget and input gates did in the LSTM.
2. Reset gate. Decides how much of the previous memory to ignore when computing the new candidate memory.
On top of that, the GRU combines the cell state and the hidden state into one. Fewer moving parts means it’s lighter, faster to train, and in many cases just as effective.
LSTM vs GRU: which is better?
It depends on the case, and in practice the difference is usually small.
| Model | Advantages | When to use it |
|---|---|---|
| LSTM | more expressive, finer control over memory | very long sequences, complex tasks |
| GRU | simpler, faster, fewer parameters | large datasets, fast training, limited resources |
The practical rule: if you have plenty of compute and a task that demands very fine-grained memory, the LSTM gives you control. If you want to train fast with less data or fewer resources, the GRU usually performs just as well or better. Many modern models, when they still use recurrence, prefer the GRU precisely for that balance.
What did they actually solve?
LSTMs and GRUs achieved something basic RNNs couldn’t:
- ✔ taming the vanishing gradient
- ✔ avoiding the exploding gradient
- ✔ remembering information over the long term
- ✔ stabilizing training
For the first time, a neural network could hold on to important information across the entire sequence without losing it along the way.
The bridge to attention and Transformers
LSTMs and GRUs were a huge leap… but they still carried underlying limitations:
- they process sequences step by step (they don’t parallelize well)
- they still depend strictly on order
- their memory, though enormous, is still limited
- they’re slow with very long texts
That’s where two ideas that changed everything came in.
Attention said:
«What if, instead of compressing everything into one memory, we look directly at what’s relevant at each moment?»
Transformers said:
«What if we process the whole sequence in parallel, instead of going word by word?»
But it’s worth remembering: without LSTMs and GRUs, attention would never have had a place to be born.
🟦 In summary
LSTMs and GRUs were the first networks capable of truly remembering: they use gates to control what to forget, what to store, and what to use, they solve the vanishing gradient problem, they capture long-range dependencies, and they were the standard in NLP for years.
Their great contribution wasn’t just technical, but conceptual:
Remembering well isn’t about clinging to everything, but about choosing what’s worth remembering.
They’re the bridge between classic RNNs and modern language models, and the last step before attention rewrote everything.