Multimodality — CLIP: when AI learned to understand images and text in the same language
The bridge that unified vision and language and gave rise to modern AI
Joint embeddings, shared latent spaces, and the conceptual engine behind Stable Diffusion, Midjourney, GPT-4o Vision, and Gemini.
For years, artificial intelligence lived split across two continents that seemed doomed never to understand each other:
- The world of language: governed by sequences, attention, and Transformers.
- The world of vision: governed first by convolutions (CNNs) and, later, by Vision Transformers (ViTs).
They were two universes speaking incompatible languages. One thought in chains of words; the other, in grids of pixels. A machine could write a brilliant essay on astronomy or classify photos of galaxies, but there was no elegant bridge connecting the meaning of both worlds. If you wanted a model to recognize a photo, you were tied to rigid, predefined labels (like ImageNet’s fixed 1,000 classes): if a new object appeared, the network fell silent.
Everything changed in early 2021, when OpenAI unveiled a model that seemed almost too simple… but that rewrote history:
CLIP — Contrastive Language–Image Pretraining.
It’s worth clarifying right away what CLIP is not, because that’s where almost all the confusion lives:
- It does not generate images (a diffusion model does that).
- It does not answer questions like a chatbot.
- It is not a traditional closed-box classifier.
CLIP is something deeper and more humble: the universal translator between sight and word. The missing «Google Translate» between pixels and letters.
1. The founding idea: a single map for two worlds
Before CLIP, each type of model drew its own map. Vision models placed images in one mathematical space; language models placed words in another, completely isolated map. They were two separate atlases, without a single city in common.
CLIP started from a hypothesis as intuitive as it was devastating:
If the photo of a sleeping cat and the written phrase
"a cat resting"describe exactly the same thing in reality… they should land at the same mathematical coordinates.
We call that meeting point the shared latent space. It doesn’t matter whether the input is pixels or characters: if the concept is the same, the resulting vector must fall in the same place. That single common map is the seed of all modern multimodality.
2. Two specialized models, one single score
To pull this off, CLIP didn’t invent anything from scratch: it joined the two architectures that already ruled each world and forced them to play the same song.
- A Vision Transformer (ViT): takes the image, slices it into patches, and turns it into a visual vector.
- A standard text Transformer: takes a sentence, computes its contextual embeddings with attention, and turns it into a textual vector.
The crucial detail: both project their output into a geometric space with the same number of dimensions (for example, vectors of 512 or 768 numbers). Without that condition, comparing an image with a text would be like trying to add pears and kilometers.
But the real magic isn’t in the encoders, it’s in how they’re trained together: contrastive learning.
3. The trick of massive contrastive learning
How do you convince two neural networks that were born separately to speak the same mathematical dialect?
OpenAI scraped a colossal catalog from the internet: 400 million image-text pairs (real photos accompanied by the descriptions already associated with them online).
During training, the model is shown an enormous batch; for example, 32,768 images alongside their 32,768 corresponding texts. And then two opposing forces are applied at the same time:
- Attraction force: the vector of image 1 must get as close as possible to text 1 (its correct match).
- Repulsion force: that same image 1 must move as far as possible from the other 32,767 texts in the batch that don’t belong to it.
It’s like subjecting the network to a card-matching game millions of times in a row. The model doesn’t tune its parameters to memorize specific pixels, but to understand the semantic correspondence between what is seen and what is read. It learns relationships, not recipes.
4. The end of fixed labels: «Zero-Shot» classification
Here is one of the most spectacular consequences. Before CLIP, classifying images was a rigid and expensive process. If you trained a model to tell cats from dogs and the next day you needed to detect an «experimental underwater drone,» the model failed hopelessly: you had to gather thousands of new photos, label them by hand, and retrain the whole network.
CLIP tore that wall down with zero-shot classification (classifying without having been specifically trained for that task).
| Classic approach (ImageNet-style) | CLIP approach (Zero-Shot) |
|---|---|
| Fixed classes (e.g. Class 42 = «Dog») | Infinite vocabulary in natural language |
| Requires thousands of labeled photos per category | Only requires writing a descriptive sentence |
| If a new object appears, retraining is needed | If the concept can be written, it can be recognized |
How does CLIP classify something it was never taught?
- You take the photo of a mysterious animal.
- You write several options in plain text, using templates (prompts):
"a photo of a cheetah""a photo of a leopard""a photo of a jaguar"
- You pass the photo through the visual encoder and each sentence through the text encoder.
- You compute the cosine similarity (basically, the geometric angle) between the image vector and each sentence vector.
- The sentence whose vector ends up closest to the image wins the match.
CLIP can recognize concepts it never saw explicitly grouped together, because during training it didn’t learn closed boxes, but general concepts of the world.
5. Language as the supreme semantic compressor
Why does something apparently so simple work so well? Because human language is the most powerful information compression system that exists. Look at this sentence:
«a small dog running on a snowy hill at sunset»
In barely 10 words, the following are compressed all at once:
- Subject: a dog, and a small one at that.
- Dynamic action: running.
- Setting and terrain: a snow-covered hill.
- Moment and lighting: the golden, low-angle light of sunset.
- Composition and intent implicit in the whole scene.
By forcing the vision network to project the image onto the exact same point as that text summary, you push the visual model to capture not just edges and colors, but the complete narrative of the scene. The text acts like a teacher summarizing for the eyes what is truly important in each photo.
6. The real impact: the spark that lit up modern AI
CLIP wasn’t just another model in a lab: it was the piece that unclogged several of today’s most disruptive technologies.
1. The generative revolution (Stable Diffusion, Midjourney, DALL·E)
A diffusion model only knows how to transform chaotic noise into sharp shapes, but on its own it is deaf: it has no idea what «an astronaut riding a horse on Mars» means. CLIP is the conductor: it translates the user’s prompt into a guiding vector within the latent space, and the diffusion model keeps sculpting the pixels until its visual vector matches that text vector.
2. Visual semantic search without labels
Modern photo search engines (Google Photos, Apple Photos) let you type «the day the cake fell» and find the exact image, without anyone having previously labeled that file. They search for nearby vectors in the shared latent space.
3. Frontier multimodal models (GPT-4o Vision, Gemini, PaLI, Flamingo)
Today’s large conversational models are no longer text processors with a vision module stuck on as a patch: their representation spaces unify text tokens with image patches (and even audio spectrograms) under the same principle CLIP pioneered.
7. CLIP in the productive world
Beyond the headlines, CLIP is the silent engine behind a great many everyday applications: smart visual search engines, automatic tagging systems in e-commerce (classifying millions of garments by style, color, and season with no human intervention), content moderation filters (detecting inappropriate material by comparing images with prohibited descriptions), multimodal assistants, analysis of documents containing images, and industrial robots capable of interpreting orders like «pick up the rusty tool from the table» by visually identifying the described object.
In every one of those cases it does the same thing: translate between what is seen and what is said.
In summary
- For years, vision and language operated in separate mathematical universes, with no bridge between them.
- CLIP unified them by projecting images and sentences into a shared latent space.
- It’s trained through massive contrastive learning (400 million pairs): pulling matching vectors together, pushing non-matching ones apart.
- It enabled zero-shot classification, saying goodbye forever to the dependence on fixed classes.
- It became the indispensable semantic engine of image generators like Stable Diffusion and Midjourney.
- It is the conceptual foundation of today’s multimodality (GPT-4o Vision, Gemini, PaLI, Flamingo).
- It proved that language is the most effective tool for teaching a machine to understand the content of an image.
It was the exact moment when artificial intelligence stopped seeing photos as cold grids of numbers… and began interpreting them as ideas.