Word2Vec (CBOW + Skip-Gram)
The model that taught machines the meaning of words
How to turn language into vectors that capture relationships, context, and semantics.
In the previous chapter we saw how Autoencoders learn to compress and reconstruct data, discovering its essence. Today we take another step: turning words into vectors that capture meaning.
Before Word2Vec, words were arbitrary numbers or one-hot vectors: huge, sparse representations with no semantics whatsoever.
To a machine, «cat» and «dog» were as different as «cat» and «astronomy.» There was no way to know that two words were related.
Word2Vec changed that forever.
The core idea: words with geometric meaning
Word2Vec turns each word into a dense, small, information-rich vector. And it does so with a simple but powerful idea:
Words that appear in similar contexts have similar meanings.
If «king,» «queen,» «castle,» «throne,» and «crown» appear together across many texts, their vectors will end up close together in space. If «dog,» «cat,» «pet,» «food,» and «vet» appear together, so will theirs.
Word2Vec learns these relationships without anyone telling it to. Just by reading text.
A way to picture it. Think of a giant map where each word is a point. At first they’re placed at random. As Word2Vec reads millions of sentences, it keeps moving each point: it pulls together words that tend to appear in the same contexts and pushes apart the ones that have nothing to do with each other. In the end, «cat» and «dog» become neighbors, while «astronomy» ends up in a different part of the map. Nobody drew that map by hand: it emerged on its own from the text.
What does Word2Vec actually do?
Word2Vec doesn’t try to classify or predict labels. Its goal is more subtle:
Learn representations that capture the context of each word.
To do this, it trains a tiny neural network on a very simple task, which it can frame in two opposite ways:
- CBOW → predict the central word from the context.
- Skip-Gram → predict the context from the central word.
And here’s the clever trick: the network itself is never the goal. What we actually care about are the weights it learns along the way. Those weights are the embeddings.
CBOW: learning from the context
CBOW (Continuous Bag of Words) works like this:
«Tell me the surrounding words, and I’ll tell you what the central word is.»
Example:
- Context: «the ___ is on the roof»
- Central word: «cat»
The network receives the context words and learns to predict the missing one. With thousands of examples, it discovers which words tend to appear together, and that creates vectors that capture semantics.
✔ Advantages: it’s fast, works well with small datasets, and is stable.
Skip-Gram: learning from the central word
Skip-Gram does exactly the opposite:
«Tell me the central word, and I’ll tell you which words tend to surround it.»
Example:
- Central word: «cat»
- Context: «the,» «is,» «on,» «the,» «roof»
The network learns which words tend to appear around «cat.»
✔ Advantages: it’s better for large vocabularies, captures more complex relationships, and produces richer embeddings.
How is Word2Vec trained?
Word2Vec uses a very simple neural network:
- Input: a one-hot vector.
- Hidden layer: the embedding.
- Output: the probability of the context words.
But training this directly would be slow: every prediction would have to be compared against the entire vocabulary (sometimes hundreds of thousands of words). That’s why Word2Vec uses two clever tricks:
- Negative Sampling. Instead of predicting every word in the vocabulary, it only updates the correct word and a few randomly chosen «negative» ones. This speeds up training enormously.
- Hierarchical Softmax. It organizes the vocabulary into a tree to drastically reduce the number of calculations.
Thanks to these tricks, Word2Vec can train on millions of words without trouble.
The magic: algebraic relationships between words
Word2Vec doesn’t just group similar words together. It does something far more surprising: it captures semantic relationships as mathematical operations.
The classic example:
Read that formula like a sentence: «take the concept of king, remove what it has of man, add what it has of woman… and you land on queen.» The machine doesn’t know what royalty or gender are; it has simply placed the vectors in such a way that this subtraction and addition point to the right spot on the map.
The «jump» from man to woman is almost the same vector as the jump from king to queen. That direction, learned on its own, is what encodes the concept of gender.
This happens because the vectors don’t just represent words, but concepts: gender, role, hierarchy, context, semantics. It was the first time a machine could «reason» about words using pure geometry.
Why was Word2Vec so revolutionary?
Because it solved three problems at once:
- Dense representations. We went from giant, empty vectors to small vectors full of meaning.
- Emergent semantics. The relationships between words appeared on their own, without supervision.
- A foundation for everything that came after. Word2Vec opened the door to GloVe, FastText, contextual embeddings, Transformers, generative models, and modern LLMs.
It was the first step toward the vector language used by all of today’s networks.
CBOW vs Skip-Gram: which is better?
It depends on the case:
| Method | Best when… | Advantages |
|---|---|---|
| CBOW | the dataset is small | fast and stable |
| Skip-Gram | the vocabulary is large | richer relationships |
In practice, Skip-Gram tends to produce more powerful embeddings.
The bridge to GloVe and modern embeddings
Word2Vec was the first model to prove that meaning can be represented as geometry.
From there, each model added a new piece:
- GloVe took that idea further by using global statistics of the text.
- FastText added subwords, so it could also understand new or misspelled words.
- Transformers added dynamic context.
- Today’s LLMs generate embeddings that change depending on the sentence each word appears in.
But it all started here.
In summary
Word2Vec is the model that taught machines the meaning of words. It learns from context, generates dense and semantic vectors, captures algebraic relationships, works without supervision, and is the foundation of all modern embeddings.
It’s the bridge between symbolic language and vector language: the first step toward understanding text as a geometric space full of meaning.