Modern tokenization (BPE, SentencePiece): how models break language into mathematical pieces

Subwords, merges, vocabularies, and the mechanism that lets LLMs understand any language, new word, or typo

The chapter that explains how models learn to «read» before they can think.

So far we’ve explored the big structures of AI: how networks process sequences, how they pay attention, how they build embeddings, and how they join vision and language. But there’s a prior step, silent and absolutely fundamental, that happens before the first neuron fires:

Before a model can reason, generate, translate, or answer, it needs to convert text into units it can process.

Computers don’t understand letters, words, or sentences: they only know how to handle numbers. The process in charge of translating the raw text you type into digestible numeric pieces is called tokenization.

It’s such a critical phase that changing a model’s tokenizer is equivalent to changing the entire alphabet of a language: if you touch that piece, the whole internal universe of embeddings the model had learned collapses instantly.

For years, three paths were tried… and two of them failed spectacularly.

---

1. The classic dilemma: the two extremes that collided with reality

To slice up a text, intuition pushes us toward two obvious options: go word by word, or letter by letter. Both have fatal traps.

Approach 1: by whole words (Word-level)

It seemed like the most natural thing in the world: split by spaces and assign a number to each word in the dictionary.

• The disaster: human language has no end. There are plurals, conjugations, diminutives, regionalisms, typos («ur welcom»), and neologisms born every day («metaverse», «tiktoker», «prompting»).
• The result: gigantic vocabularies of millions of terms that blow up memory consumption and slow everything down. And worst of all: if the user writes a word that wasn’t on the fixed list, the model goes completely blind and spits out the dreaded[UNK]token (Unknown).

Approach 2: by individual characters (Char-level)

The opposite solution: make each letter or punctuation mark a token.

• The advantage: the vocabulary is tiny (just a few dozen symbols) and there are never unknown words.
• The disaster: sentences become ridiculously long. Watch how the tokens explode:
• «transformers» → 12 tokens
• «transformadores» → 15 tokens
• «anticonstitucionalmente» → 23 tokens

Since Transformers have a strict limit on their context memory, you’d spend half of their capacity just spelling out a few sentences before you could even reason about them.

The conclusion was obvious: a middle ground was needed. Neither whole words, nor loose letters. Something more flexible, more compact, and smarter.

---

2. The revolution: subword tokenization

The idea that changed everything was to split language into recurring fragments: pieces that show up again and again across many different words.

The solution: slice words into frequent subwords.

The LEGO brick analogy. Instead of manufacturing a pre-assembled toy for every object in the universe (whole words) or giving you only loose plastic atoms (characters), you’re handed a smart set of medium-sized blocks: common prefixes, roots, and suffixes.

Look how natural it feels:

• «transformation» →"trans"+"form"+"ation"
• «impossible» →"im"+"possible"

If a word never seen during training appears, the model doesn’t panic or go dark: it rebuilds it on the fly by combining pieces it already knows.

---

3. BPE (Byte-Pair Encoding): the craftsman that fuses pieces

BPE is the algorithm that conquered modern language processing (used from GPT-2 to today’s models). The curious part is that it was born in the 1990s as a simple data compression algorithm… and turned out to be the perfect tool for text.

How does BPE training work?

1. Starting point: it begins by treating each character as an independent token.
2. Counting: it examines millions of texts looking for which pair of adjacent tokens appears together most frequently.
3. Merge: it joins that winning pair to create a new token and adds it to its vocabulary.
4. Repetition: it repeats the cycle thousands of times, until reaching the set vocabulary size (for example, 50,000 tokens).

A step-by-step example

Imagine a corpus where the words «transform», «transformation», and «transformers» repeat a lot. BPE will keep merging by frequency:

At the end of the process, the algorithm has learned – purely statistically, without knowing anything about human grammar – that"trans"and"form"are very powerful units of meaning, and it leaves them as solid blocks. It’s a craftsman who builds the language from the bottom up, welding together the pieces it most often sees side by side.

---

4. WordPiece vs. BPE: two ways of thinking

Historic models like BERT introduced a variant called WordPiece. The goal is identical (create subwords), but its philosophy for deciding which pieces to join is different:

In short: BPE is the mechanical fuser; WordPiece is its academic cousin, which instead of counting co-occurrences calculates which merge makes the text most predictable.

---

5. SentencePiece: the definitive multilingual standard

Despite BPE’s success, classic tokenizers carried a very Western vice: they assumed words always come separated by whitespace.

That works in English or Spanish, but it breaks down completely with languages like Chinese or Japanese, where there are no spaces between words:

$\text{我喜欢人工智能}\text{(I like artificial intelligence)}$

In 2018, Google released SentencePiece, the piece of engineering used today by Llama, T5, PaLI, and Gemini, among many other multilingual models.

Its key leaps

1. It treats text as a raw sequence of bytes: you don’t need to clean anything beforehand. It accepts emojis 🚀, math symbols, code, hashtags, and broken URLs without blinking.
2. The space is a visible character: it replaces spaces with a special symbol (usually▁). This allows it to de-tokenize – rebuild the original text – exactly and reversibly, without losing a single comma or line break.
3. Real multilingualism: it processes non-Latin alphabets and languages without word separation as smoothly as English.
4. Flexible: it can operate in BPE mode or Unigram mode, depending on what suits the model.

In practice, it’s the most robust tokenizer in the real world.

---

6. A real comparison: the journey of a complex word

Watch how each approach processes a technical, unusual word like «anticonstitucionalmente» (Spanish for «unconstitutionally»):

The model grasps the meaning of the compound word without ever having seen it whole during its training.

---

7. Why does tokenization matter so much (and hit your wallet)?

Tokenization isn’t a lab-only technical debate: it has direct economic and performance consequences.

• Inference cost (your API bills): OpenAI, Anthropic, and Google don’t charge by words or by letters; they charge per million tokens. If a bad tokenizer splits a text into 1,500 tokens instead of 1,000, you’re paying 50% more to process exactly the same information.
• The «non-English tax» (language bias): historically, LLM vocabularies were saturated with English subwords. While an English word cost 1 token, its translation into Spanish or German required 2 or 3, and in Arabic or Hindi it could spike to 6 tokens per word. Modern tokenizers with large vocabularies (Llama 3’s or GPT-4o’s 128k tokens) have corrected that gap, making usage cheaper in other languages.
• Context capacity: your model has an attention limit (8k, 32k, 128k tokens…). An efficient tokenizer compresses text better, which lets you fit longer books or denser code into the same space.
• Model quality: embeddings depend entirely on the tokenizer. Changing it midway destroys the semantic space the model had learned with so much effort.

---

In summary

Modern tokenization solved the founding puzzle of how to make machines interpret our language:

• It discarded the extremes: it overcame the blindness of whole words and the suffocating slowness of loose characters.
• It adopted subwords: it built flexible vocabularies with reusable LEGO pieces, able to reconstruct any new word.
• BPE learned to merge: it finds the most common roots and patterns based on the statistical frequency of texts.
• SentencePiece erased the borders: it took tokenization down to the byte level, working transparently with any language on the planet, code, or emojis.
• It dictates efficiency: it defines how much context fits in memory and how much it costs to run the model.

It was the exact moment when machines stopped reading text as a chaotic alphabet soup… and began building the pieces of meaning with it.