Basic RNNs + Bidirectional RNNs
The networks that tried to remember… before attention changed everything
How RNNs work, why they were revolutionary, and how the bidirectional versions expanded their ability to understand sequences.
So far we’ve looked at models that compress (Autoencoders) and models that capture geometric meaning (Word2Vec, GloVe).
Today we step into a different family: networks that process sequences.
Before Transformers, before attention, before LLMs… RNNs were the first serious attempt at getting a neural network to remember.
The core idea: processing information step by step
An RNN (Recurrent Neural Network) is a network built to work with data where order matters:
- text
- audio
- time series
- sensor signals
The intuition is simple:
An RNN processes the sequence element by element, keeping an internal state that summarizes everything it has seen so far.
That state is its memory.
One way to picture it. Think about how you read this sentence. You don’t understand each word in isolation: you carry what you’ve already read and use it to interpret what comes next. By the time you reach the period, your understanding is the sum of the whole journey. An RNN does exactly that: it reads, updates its idea, and moves on.
What makes an RNN different?
A regular network takes an input, produces an output, and that’s it. An RNN adds one key ingredient: it passes information to itself from one step to the next.
At each step it does four things:
- Receives one element of the sequence (a word, a number, a vector).
- Updates its internal state by combining the current input with what it already remembered.
- Produces an output (optional).
- Moves to the next element, carrying its memory along.
That internal state is fed back at every step. That’s why it’s called recurrent.
Notice the chain of memories: each state inherits from the previous one. That arrow running from $h_1$ to $h_2$ to $h_3$ is all the magic (and, as we’ll see, also all the trouble).
What happens mathematically?
At each step $t$, the new memory is computed like this:
Where:
- $x_t$ is the current input
- $h_{t-1}$ is the previous memory
- $h_t$ is the new memory
- $W$ and $U$ are weight matrices the network learns
And the output, when there is one, comes from that memory:
It’s a simple mechanism… but a surprisingly powerful one. All the «intelligence» lies in how it blends the new ($x_t$) with the old ($h_{t-1}$).
What can RNNs do?
With that step-by-step memory, RNNs opened the door to tasks that used to be very hard:
- predicting the next word
- classifying sequences
- generating text
- sentiment analysis
- modeling time series
- detecting patterns in sequential data
They were the foundation of the first language models.
The problem: remembering is hard
Basic RNNs have a serious flaw:
They struggle enormously to hold on to information across many steps.
The reason is technical, but it’s easy to grasp with an image. When the network learns, it has to «propagate» the error signal backward through the entire chain of memories. And when you multiply many times over, one of two things happens:
| Problem | What happens | Consequence |
|---|---|---|
| Vanishing gradient | gradients shrink to almost nothing | the network forgets what happened at the start |
| Exploding gradient | gradients blow up | training becomes unstable |
An example of why it hurts. Imagine the sentence:
«I was born in France… (thirty words later) …that’s why I speak fluent ______.»
To fill in the blank with «French,» the network has to remember the first word thirty steps back. A basic RNN has probably already forgotten it. That’s why they work well with short sequences but fail with long sentences or complex contexts.
Bidirectional RNNs: looking forward and backward
A normal RNN only sees the past. But in language, the future matters too.
Look at this example:
«I sat on the bank to rest.»
«I went to the bank to withdraw money.»
The word bank means different things, and the clue isn’t before it, but in the words that come after. An RNN that only reads left to right reaches «bank» without knowing yet which of the two meanings applies.
To solve this, bidirectional RNNs (BiRNNs) were born.
How does a bidirectional RNN work?
A BiRNN doesn’t have one network, but two:
- one that processes the sequence forward (→)
- another that processes it backward (←)
Then it combines both memories for each word.
It’s like reading the sentence twice: once from left to right and once from right to left. That way, each word is understood with both past and future context at once.
Why were they so important?
BiRNNs were a huge leap because they:
- capture the full context of each word
- handle ambiguities far better
- perform better on classic NLP tasks
- were the foundation of models like ELMo
Before Transformers, BiRNNs were practically the standard for understanding language.
RNN vs BiRNN: clear differences
| Model | What it sees | Advantages |
|---|---|---|
| RNN | only the past | simple and fast |
| BiRNN | past + future | better semantic understanding |
The practical rule: if you need the whole sentence before deciding (translation, sentiment analysis, tagging), the BiRNN wins. If you have to predict in real time, word by word, without knowing the future (like generating text), you can only use the forward version.
The bridge to LSTM, GRU, and Transformers
Basic RNNs were the first attempt at memory. BiRNNs expanded the context. But both still dragged along the gradient problem.
That’s why improvements kept appearing:
- LSTM → memory with gates that decide what to remember and what to forget
- GRU → a simpler, lighter version of the LSTM
- Attention → dynamic memory that looks directly at what’s relevant
- Transformers → full parallelization and global context
RNNs are the origin of it all. Transformers, the final evolution.
In summary
RNNs were the first model able to process sequences with memory: they update an internal state, remember what they’ve seen, and produce outputs step by step.
Bidirectional RNNs expanded that memory by looking forward and backward, capturing the full context and improving language understanding.
But they also left us a lesson:
Remembering over the long term is hard. And that very difficulty is what drove everything that came afterward, all the way to Transformers.