Sampling and decoding: how an LLM decides what to say
Why two models that «know» exactly the same thing can give you completely different answers
We’ve already seen how a language model reads (tokenization), how it stores meaning (embeddings), how it connects ideas (attention), and how it guesses the next word.
But one question looks simple and really isn’t:
If a model only computes probabilities… who actually decides the word it ends up writing?
The answer is stranger than it seems. Because a model’s knowledge doesn’t change from one answer to the next: what changes is how it chooses. And that way of choosing is, deep down, what gives it «character.»
One line worth keeping:
Knowledge lives in the model’s weights. Personality is born in how it picks its words.
1. The moment of decision
Imagine we type:
«The sky is…»
The model doesn’t reply with a single word. It replies with a list of candidates and their probabilities, something like:
| Word | Probability |
|---|---|
| blue | 62 % |
| clear | 17 % |
| beautiful | 8 % |
| dark | 5 % |
| cloudy | 4 % |
| the rest | 4 % |
This is what’s called a probability distribution. Notice the key detail: the model doesn’t know the correct answer. It only says:
«Based on everything I read during training, this is the most likely thing to come next.»
From here on, the real star of this post takes over: the algorithm that turns that list into one concrete word.
2. Greedy: the student who never takes a risk
The simplest strategy in the world:
Always pick the most likely word. Full stop.
In our example: The sky is → blue. Then start over with the next word.
It’s fast, predictable, and easy. Feed it the same sentence ten times and you get the same answer ten times.
The problem? It’s hopelessly short-sighted. It only looks at the next step, never at whether that choice will wreck the sentence later on. It’s like playing chess thinking only about the move right in front of you, ignoring what comes after. Sometimes the best word now leads to a dead end later.
3. Beam Search: don’t marry your first idea
Beam Search is smarter: instead of committing to a single option, it keeps several candidate sentences open at once and sees which one works best overall.
With a «width» of 3, for example, it keeps exploring three paths in parallel:
- The sky is blue…
- The sky is clear…
- The sky is beautiful…
In the end it keeps the complete sentence that fits best, not the best single word.
It works beautifully when there’s a «correct» answer to find: translating a text, summarizing a document. But there’s a catch: it tends to be bland and repetitive, because it always plays it safe. That’s why chatbots that want to sound natural almost never use it.
4. Temperature: the creativity thermostat
Here’s where it gets fun. Temperature doesn’t change what the model knows; it changes how much it gambles when it chooses.
The idea in one line: do you want a cautious model or a bold one?
Low temperature (e.g. 0.1) → exaggerates the gaps. The favorite becomes almost unstoppable:
| Word | Before | With low T |
|---|---|---|
| blue | 62 % | 94 % |
| clear | 17 % | 4 % |
| beautiful | 8 % | 1 % |
Result: safe, predictable answers.
High temperature (e.g. 1.5) → flattens the gaps. The underdogs get a real shot:
| Word | With high T |
|---|---|
| blue | 40 % |
| clear | 22 % |
| beautiful | 18 % |
| dark | 12 % |
Result: more varied and surprising answers (and sometimes more nonsensical, too).
The analogy: temperature is the risk dial. Turn it down and you get a meticulous engineer; turn it up and you get a writer with a runaway imagination.
5. Top-k: keep only the best candidates
Another simple, effective idea:
Throw out everything except the top k options, and choose only among those.
With Top-k = 3 we keep blue, clear, and beautiful, and toss out the thousands of absurd options with a tiny probability.
The model still has room to surprise you, but without suddenly blurting out a word that makes no sense. Creativity, but with a safety net.
6. Top-p (nucleus sampling): today’s standard
Top-k has one weakness: fixing a set number of candidates doesn’t always fit. Sometimes 3 options are too few; other times half of them are too many.
Top-p solves this more cleverly:
It doesn’t fix a number of candidates, it fixes a percentage of accumulated confidence.
With Top-p = 0.9, the model adds up candidates (from most to least likely) until it reaches 90 % probability, and discards the rest:
| Word | Probability | In? |
|---|---|---|
| blue | 62 % | ✅ |
| clear | 17 % | ✅ |
| beautiful | 8 % | ✅ |
| dark | 5 % | ✅ (totals 92 %) |
| cloudy | 4 % | ❌ |
The upside: the candidate group adjusts itself to the context. When the model is very sure, it considers few options; when it hesitates, it widens the field. That’s why Top-p is the dominant method today in modern models.
7. Repetition penalty: breaking the loops
An uncomfortable classic:
The cat was sleeping.
The cat was sleeping.
The cat was sleeping.
Models love repeating patterns. The repetition penalty prevents this with a simple rule: every time a word has already appeared, it loses points and becomes less tempting. That nudges the model to look for new paths instead of going in circles.
8. The idea that changes everything: models don’t write, they bet
This is the «magic trick» behind it all. An LLM never decides:
«I’m going to say this.»
What it actually does is:
«These are the probabilities of every word that could come next.»
And it’s the decoding algorithm (greedy, temperature, top-p…) that turns that bet into a sentence. The model lays out the probabilities; the strategy makes the decision.
9. Why the same GPT can seem like different people
Because you don’t need to change what it knows: you just turn a couple of dials.
| Setting | How it sounds |
|---|---|
| Temperature 0 | Precise, but repetitive |
| Temperature 0.7 | Balanced |
| Temperature 1.2 | Creative |
| Low Top-p | Cautious |
| High Top-p | Exploratory |
The same model, without touching a single weight, can sound like a teacher, a programmer, a writer, or a poet. Only the way it picks its words changes.
In short
LLMs don’t hand you an answer directly. First they build a probability distribution for the next word. Then a strategy decides which path to take:
- Greedy → always the most likely option
- Beam Search → explores several sentences at once
- Temperature → turns the risk level up or down
- Top-k → keeps the best candidates
- Top-p → adjusts the candidates to the context
- Repetition penalty → avoids loops
This was exactly where AI stopped merely predicting words… and started to develop something that, from the outside, looks a whole lot like having a style of its own.