GloVe + Bias in Embeddings
When meaning emerges from statistics… and so do the biases of the real world
How GloVe learns global language relationships, and why embeddings can inherit human prejudices.
In the previous chapters we saw how Autoencoders learn to compress the essence of data, and how Word2Vec turns words into vectors that capture context and semantics.
Today we get to a model that took that idea one step further: GloVe. And also a crucial topic: the biases that live inside embeddings.
Because if machines learn from the world… they also learn its prejudices.
GloVe: Global Vectors for Word Representation
Word2Vec learns meaning by looking at local contexts: windows of a few words around each term. GloVe does something different: it looks at the global statistics of the entire text all at once.
The core idea is elegant:
If two words appear together very often, that relationship should be encoded in their vectors.
But GloVe doesn’t stop there. It doesn’t just look at whether they appear together, but how much more they appear together compared to other combinations. It’s a model that learns from the statistical balance of language.
One way to picture it. Word2Vec is like someone who learns by reading sentence by sentence, with a flashlight that only lights up the neighboring words. GloVe, by contrast, is like someone who first counts the entire book, builds a big table of «who appears with whom,» and only then deduces meanings from those totals.
The intuition: meaning as ratios
Imagine you analyze millions of sentences and observe these four facts:
- «ice» appears near «cold» very often
- «steam» appears near «hot» very often
- «ice» appears near «hot» very rarely
- «steam» appears near «cold» very rarely
GloVe observes these ratios and concludes, without any hand-written rule:
«ice is more related to cold than to hot»
«steam is more related to hot than to cold»
The subtle trick lies in the ratios, not the raw counts. Notice what happens when you compare the two words against a third one (the numbers are illustrative):
| Probe word | Appears with «ice» | Appears with «steam» | Ratio ice/steam |
|---|---|---|---|
| solid | high | low | ≫ 1 → belongs to ice |
| gas | low | high | ≪ 1 → belongs to steam |
| water | high | high | ≈ 1 → common to both |
| fashion | low | low | ≈ 1 → irrelevant to both |
The ratio distinguishes what truly separates two words (solid vs gas) from what they share, or from what simply doesn’t matter. That signal is exactly what GloVe tries to carve into the geometry of the vectors.
It’s as if the model were saying:
«If two words co-occur much more than expected, they must be semantically connected.»
How does GloVe work?
GloVe starts by building a global co-occurrence matrix of the whole corpus:
- rows → words
- columns → words
- values → how many times they appear together
Then GloVe tries to factorize that matrix to obtain dense vectors that reproduce those relationships. Its objective function is based on a simple idea:
The difference between two vectors should reflect the ratio of co-occurrences between their words.
Put another way, it wants the dot product of two vectors to match the (logarithm of the) number of times those words appear together:
where $X_{ij}$ is how many times word i appears next to word j. This lets GloVe learn analogies, similarities, semantic directions, and abstract concepts, all without depending on Word2Vec’s local windows.
Why was GloVe so important?
Because it combined the best of two worlds:
- Global corpus statistics. It captures relationships Word2Vec misses because they fall outside the local window.
- Dense, algebraic vectors. It keeps the geometric properties that made Word2Vec famous (that «word arithmetic» from the previous chapter).
- More stable analogies. Relationships like «king − man + woman ≈ queen» tend to come out more consistent.
- A foundation for what came next. GloVe influenced FastText, contextual embeddings, Transformers, and generative models.
Word2Vec vs GloVe: what’s the difference?
| Model | Learns from… | Strengths |
|---|---|---|
| Word2Vec | local contexts | fast, efficient, very practical |
| GloVe | global statistics | more stable analogies, broader relationships |
In practice, both produce excellent embeddings. The difference is one of approach: Word2Vec learns «on the fly» by reading windows, while GloVe learns from the full snapshot of the corpus. GloVe tends to capture global language relationships better.
Bias in embeddings: the dark side of meaning
So far it all seems magical: vectors that capture semantics, relationships, analogies… But there’s a problem.
If the text of the world has biases, embeddings will have them too.
And the text of the world does have biases. This is the flip side of the coin: the very mechanism that makes embeddings so powerful (learning from the statistics of real language) is the same one that makes them absorb the prejudices in that language.
What kinds of biases show up?
Embeddings can learn, among others:
- Gender bias: «doctor» closer to «man,» «nurse» closer to «woman.»
- Racial bias: unfair associations between groups and negative adjectives.
- Cultural bias: professions, roles, and adjectives tied to certain communities.
- Historical bias: words that reflect past inequalities.
The model doesn’t invent these biases. It inherits them from the text. If you train on millions of web pages, you learn what the internet thinks. And that includes prejudice.
Why does this happen?
Because embeddings capture the statistics of language, and language reflects culture, history, inequality, stereotypes, and social bias. Models have no morals: they just learn patterns. If a pattern is in the data, it ends up in the vectors.
How are biases detected?
A classic technique is to measure semantic directions in the vector space. Just as there is a «man → woman» direction, you can trace axes like «career → family» or «science → arts.»
The test is simple: you project a word onto that axis and see which side it falls on.
If words that should be neutral (like a profession) align too strongly with one end of the axis, there’s bias. The most cited historical example:
«programmer» lands closer to «man» than to «woman.»
The model didn’t invent that. It learned it from the text.
How are biases mitigated?
There are several techniques:
- Geometric debiasing: adjusting or neutralizing the problematic directions in the vector space.
- Regularization during training.
- Retraining with balanced data.
- Corpus filtering.
- Neutral embeddings for words that shouldn’t carry gender or group loading.
But no solution is perfect. You can touch up the geometry and still leave traces of the original bias. In the end it’s an uncomfortable reminder: bias is a social problem, not just a technical one.
The bridge to modern embeddings
GloVe and Word2Vec were the first to show that meaning can be represented as geometry. From there, each model added a piece:
- FastText added subwords, to also understand new or misspelled words.
- Transformers added dynamic context.
- Modern models generate embeddings per token, not per word, with meaning that shifts depending on the sentence.
- And techniques for partial bias mitigation emerged.
But all of them rest on the intuition that was born here.
In summary
GloVe is the model that took embeddings to the next level: it learns from global statistics, captures broad relationships, produces stable analogies, complements Word2Vec, and is the foundation of modern models.
But it also taught us something important:
Embeddings don’t just capture meaning. They also capture the biases of the world.
Understanding this is essential to building responsible AI.