Vanishing / Exploding Gradient + Beam Search

The problems that made networks suffer… and the technique that let them generate text sensibly

Why training deep networks was so hard and how Beam Search helped produce coherent sequences before Transformers.

In previous chapters we saw how RNNs, LSTMs, and GRUs tried to remember over time. But even with those improvements, training deep networks was still a minefield.

Today we bring together two key pieces from that era, which act at different moments:

1. Vanishing / Exploding Gradient → the obstacle during training, which stopped networks from learning.
2. Beam Search → the strategy during generation, which made it possible to produce sentences that made sense without collapsing in the attempt.

They belong to different stages (training vs. generating), but together they formed the backbone that held up language models before attention arrived.

---

Vanishing Gradient: when the signal fades out

For a network to learn we use backpropagation: we compute the error at the end and send it backward, multiplying derivatives layer by layer via the chain rule.

The drama shows up in deep networks or long sequences: each step backward means multiplying by a scaling factor. If those factors are smaller than 1 (very common with functions likesigmoidortanh, whose derivatives rarely exceed 0.25), this happens:

$0.5\times 0.5\times 0.5\times \cdots \times 0.5\approx 0$

After just a few layers, the value is so tiny that the learning signal disappears completely.

The broken-telephone analogy. Imagine shouting a message across 30 rooms, but each person repeats it at half the volume of the previous one. By the fifth room it’s a whisper; by the first, nobody has a clue what was supposed to be fixed.

What causes it and what are its consequences?

The usual culprits aresigmoid/tanhactivations, repeated multiplications in long sequences, weights initialized too small, and deep architectures with no memory. The result is always the same:

• Early amnesia: the first steps of the sentence never update their weights; the network only remembers the very last thing it just read.
• Frozen training: even when the network makes glaring mistakes, the deep layers stay static.

---

Exploding Gradient: when the signal blows up

The opposite phenomenon happens. What if the factors are greater than 1?

$2\times 2\times 2\times \cdots \times 2=\text{Astronomical numbers}$

Within just a few steps, the gradients grow exponentially until they overflow memory.

Following the analogy: each room repeats the message at double the volume. Very soon there’s no message, just a roar that blows out the speakers.

Direct consequences:

• Weights update in giant, chaotic jumps.
• The loss function oscillates out of control or throws the dreadedNaN(Not a Number).
• The model falls apart and never converges.

---

The first-aid kit: how was it solved?

The community designed standard fixes that we still use today:

Together, these techniques finally made it possible to train deep and sequential networks with stability.

---

What does this have to do with Beam Search?

Let’s imagine we’ve dodged the runaway gradients and now have a well-trained model. Time to put it to work writing: a translation, a summary, a sentence.

Here came the second bottleneck: how does the model decide which word to place next?

1. Greedy Search: always pick the word with the highest immediate probability.
   Problem: it’s short-sighted. A word that looks good now can drag the model into a dead end three words later.
2. Exhaustive search (brute force): compute every possible combination of sentences.
   Problem: an unmanageable combinatorial explosion ($V^T$, where $V$ is the vocabulary size and $T$ the text length).

To solve that dilemma, Beam Search appeared.

---

Beam Search: exploring paths sensibly

Beam Search is the smart middle ground: instead of keeping a single option (Greedy) or trying them all (brute force), it holds in parallel a select group of the $k$ most probable hypotheses at each step.

That parameter $k$ is known as the Beam Width.

A step-by-step example ($k = 2$)

Imagine the model starts writing after the phrase: «The cat…»

1. Step 1: the model computes the probabilities of the next word and keeps only the top 2:

• Hypothesis A: «climbed» ($p=0.6$)
• Hypothesis B: «slept» ($p=0.3$)

2. Step 2: from each of those 2 options, it predicts the next words and computes the cumulative joint probability.
3. Step 3: out of the 4 resulting branches, it again keeps only the global Top-2:

• 1st best: «The cat climbed the tree» ($0.42$)
• 2nd best: «The cat slept all afternoon» ($0.24$)

4. It discards the rest and repeats until reaching the end-of-sequence token (<EOS>).

In the end, the sentence with the highest cumulative probability as a whole wins, which allows it to correct decisions that looked promising at first but later made no sense.

---

Why was it so important?

Because it made it possible to generate coherent text, improve machine translation, avoid absurd repetitions, and explore alternatives without falling into a combinatorial explosion. Before Transformers, Beam Search was the standard in sequence generation.

---

Is Beam Search still used in the age of LLMs?

Yes, but the landscape has changed depending on the task:

• Deterministic tasks (translation, technical summaries, audio transcription): Beam Search is still very popular (for example, in Whisper-style systems) because it seeks the most faithful and probable sequence.
• Conversational and creative models (GPT-4, Claude, LLaMA): here Beam Search tends to produce flat, repetitive text. That’s why modern LLMs prefer stochastic methods:
• Temperature: adjusts the level of daring when sampling.
• Top-k and Top-p (Nucleus Sampling): pick words at random only within the group of reasonable candidates, producing more human-like and varied text.

---

In summary

• Vanishing Gradient faded out the learning signal by multiplying tiny derivatives across long sequences.
• Exploding Gradient shot the numbers up until training collapsed.
• Tools like Gradient Clipping, ReLU, LSTM/GRU, and LayerNorm managed to tame those gradients.
• Beam Search solved the other half of the problem: how to generate logical sequences by exploring several alternatives at once without running out of memory.

Both advances stabilized the training and the inference of sequences, and cleared the path for the next big leap: dropping recurrence altogether and giving way to the mechanism of pure attention.