Backpropagation
The algorithm that taught networks to learn
How an idea as simple as «propagate the error backward» unlocked all of modern Deep Learning.
In the previous chapter we saw how the MLP —that chain of connected perceptrons— can learn patterns a single perceptron could never capture. But we left an open, almost uncomfortable question:
How do you adjust all those weights?
A perceptron has a handful of parameters. An MLP has thousands. A deep network, millions.
When the network gets it wrong, how does each neuron know how much to correct? How do you spread the blame for the error across millions of parameters? How do you decide which weight to raise, which to lower, and by how much?
The answer is one of the most influential algorithms in the history of machine learning.
Backpropagation.
The problem: learning isn’t trivial
Picture a network with three hidden layers. The input passes through the first layer, then the second, then the third, and finally reaches the output.
Each layer transforms the data. Each neuron applies a function. Each weight shapes the final result.
When the network fails, the error shows up at the output. But the weights that caused that error are scattered all across the network.
It’s like trying to fix a defect in a factory with hundreds of machines connected in a chain: the final part came out wrong, but… which machine went off? The first? The second? All of them a little bit?
We need a way to trace the error backward: to figure out how much each weight contributed to the failure and adjust each one in the right direction.
That’s Backpropagation.
The core idea: propagate the error backward
Backpropagation does something conceptually simple:
- It computes the error at the output.
- It spreads it backward, layer by layer.
- It calculates how much each weight contributed to that error.
- It adjusts each weight in the direction that reduces the error.
It’s a chain of responsibilities. A mathematical audit. A distributed «who’s to blame?» across the entire network.
And the magic is in how that blame gets distributed: using chained derivatives. The famous chain rule.
The chain rule: the heart of the algorithm
When a network has several layers, the output depends on the input through a composition of functions:
If you want to know how the output changes when you move a weight in the first layer, you have to differentiate a function nested inside another, which is nested inside yet another. The chain rule tells you exactly how to do it:
It’s like following a trail: each derivative tells you how a small change propagates through the network, link by link.
Backpropagation is nothing more than applying this rule efficiently, reusing calculations instead of repeating them.
An example to make it click. Think of a recipe with three steps: you chop, you cook, you season. The final dish comes out too salty. To fix it, you don’t randomly change all three steps: you trace backward and discover the problem was in the last step, the salt. The chain rule does exactly that, but measuring precisely how much «blame» each step carries for the final taste.
The full cycle: how a neural network learns
Training a network always follows the same four-phase cycle:
1. Forward pass. The input flows forward. Each layer transforms the data. The network produces a prediction.
2. Error calculation. You compare the prediction against the real label and get a loss: a measure of how far off it was.
3. Backward pass (Backpropagation). The error propagates backward. Each weight receives its share of blame. The gradients are computed.
4. Weight update (Gradient Descent). Each weight is nudged a little in the direction that reduces the loss.
And you repeat. Thousands of times. Millions of times. Until the network learns.
The intuition: teaching by correcting
Backpropagation is, at its core, a patient teacher:
- it looks at the final error,
- it distributes it among those responsible,
- it corrects each one a little bit,
- and it starts over.
Again and again.
It’s incremental learning. Small adjustments that, when accumulated, end up creating models capable of recognizing faces, translating languages, describing images, writing text, and predicting sequences.
Without Backpropagation, modern Deep Learning wouldn’t exist.
Why was it so revolutionary?
Because it solved three problems at once:
1. Learning in deep networks. Before, nobody knew how to adjust the weights of hidden layers. Backpropagation made it possible.
2. Scaling to millions of parameters. The algorithm is efficient: it uses practically the same operations as the forward pass, just in reverse.
3. Working with any architecture. MLP, CNN, RNN, Transformers… they all learn thanks to Backpropagation.
It’s the engine that drives all of Deep Learning.
In short
Backpropagation is the algorithm that taught networks to learn:
- it propagates the error backward,
- it calculates each weight’s responsibility,
- it adjusts each parameter with the gradient,
- and it repeats until the network masters the pattern.
It’s the missing piece. The bridge between the MLP and modern Deep Learning. The mechanism that let networks stop being a pretty concept and become a real tool.
And now that we know how they learn…
In the next chapter we’ll see how to keep them from forgetting, saturating, or overfitting: Regularization + Transfer Learning.