Vision Transformers (ViT): when images learned to speak the language of Transformers

Patches, embeddings, global attention, and the leap that changed computer vision

How the architecture that challenged the reign of CNNs works, and how it threw the doors of modern multimodality wide open.

For nearly a full decade (2012–2020), Convolutional Networks (CNNs) ruled computer vision with an iron fist. They were fast, efficient, elegant, and deeply intuitive.

But they carried a structural limitation from birth:

CNNs can only see the world through small local windows.

As we saw in the previous chapter, a convolution works like a small flashlight: it lights up a tiny tile of pixels (3×3 or 5×5), detects an edge or a texture, and needs dozens of stacked layers for that local information to gradually assemble a global image in the network’s mind.

In 2017, Transformers proved in the world of text that attention doesn’t need to move step by step: it works like the floodlights of an entire stadium, lighting up and interconnecting the whole scene at once.

So the bold question was only a matter of time:

What if we stop treating images as matrices of pixels and read them as if they were sentences?
What if each piece of an image were, simply, a «word»?

In 2020, the Google Research team published a paper whose title was already a statement of intent: «An Image is Worth 16×16 Words.» With it, the Vision Transformer (ViT) was born, and it triggered a conceptual earthquake in artificial intelligence.

---

1. The founding idea: slicing the photo into «visual words»

A classic Transformer demands a one-dimensional sequence (an ordered list of words: 1, 2, 3…). A photograph, on the other hand, is a two-dimensional grid of pixels (height × width × color channels).

Why not just feed in the pixels one by one and be done with it? Because it would be computational suicide. A standard 224×224 pixel photo has more than 50,000 pixels, and the cost of attention grows quadratically with the number of tokens ($O(N^2)$). Processing 50,000 tokens would blow up the memory of any GPU.

The solution was as simple as it was elegant: turn the image into a jigsaw puzzle.

The process has three steps:

1. The image is divided into a fixed grid of patches (typically 16×16 pixels).
2. For a 224×224 image, this produces exactly:
   16224​×16224​=14×14=196 patches
3. Each patch (which contains $16 \times 16 \times 3 = 768$ numeric color values) is flattened into a list and projected, via a mathematical matrix, into a vector of fixed dimension $D$.

The result is 196 vectors. And here’s the decisive mental trick: to the model, those 196 image pieces are exactly the same as a 196-word sentence. The image has stopped being a rigid grid of pixels; now it’s a visual text the Transformer can read from beginning to end.

---

2. Embeddings and Positional Encoding: where does each puzzle piece go?

A pure Transformer suffers from «spatial amnesia»: it processes all inputs at once, in a disordered bag. If you shuffled the 196 patches of a cat photo, pure attention wouldn’t notice the difference, and wouldn’t know whether the ear is on the head or stuck to the paw.

To solve this, ViT inherits two tricks directly from language processing (NLP):

1. Patch Embedding (what’s in the patch): the vector that describes the visual content of the piece (colors, edges, textures).
2. Positional Encoding (where the patch is): a numeric vector encoding its coordinates in the puzzle is added to each patch (patch 1: top left; patch 196: bottom right). Thanks to this, the model preserves the geometry and topology of the image.

Also, just as BERT did in text, a special fictional token called[CLS]is added to the start of the sequence. Think of it as a «secretary» that contributes no content of its own, but listens to the 196 patches through attention, takes notes on everything, and in the end is the only one connected to the output layer to answer the question: is this a dog, a car, or a plane?

---

3. Global attention: the superpower vision was missing

Once the patches enter the Transformer Encoder, the real leap over CNNs happens: immediate global attention (self-attention).

In a traditional CNN, the first layer can only compare a pixel with its immediate neighbors. For the signal from the top-left corner to manage to «talk» with the bottom-right one, it has to travel through dozens of intermediate convolutional layers, losing detail along the way.

In a ViT, from Layer 1, any patch can dialogue directly with any other patch in the image, with no intermediaries.

This lets the model, in a single step:

• relate both eyes to each other to measure a face’s symmetry,
• understand the complete outline of a car even when a lamppost covers the center,
• connect a tumor with the edge of the healthy tissue surrounding it,
• and instantly separate the main object from the blurred background.

---

4. Why did ViT surpass CNNs? The question of inductive bias

CNNs dominated for so long because they come with a built-in biological «trick» from the factory: inductive bias. By design, they assume two rules of the physical world:

• Locality: nearby pixels usually belong to the same thing.
• Translation invariance: an apple is still an apple whether it’s in the corner or the center of the photo.

That bias is like giving the model a free instruction manual: it learns extremely fast even with little data.

ViT, on the other hand, assumes absolutely nothing. It doesn’t know in advance what 2D space is, nor that adjacent pixels are related. It arrives in class with a blank mind and has to learn physics and visual geometry from scratch, by seeing millions of examples.

---

5. The 2020 shock: an insatiable thirst for data

When Google unveiled ViT, the initial results baffled the whole community:

• Trained on ImageNet (1.3 million photos): ViT performed worse than a conventional ResNet.
• Trained on JFT-300M (300 million private Google photos): ViT crushed any existing CNN.

The moral? Because it has no preconfigured biases, ViT needs millions and millions of examples just to intuit how the visual continuity of the world works. But once it crosses that data threshold, it no longer has the rigidity that limits a CNN. It’s an architecture born by and for massive scale.

ViT is an architecture of scale: the more data you give it, the more it shines.

---

6. The evolution: democratization and hybrid architectures

The requirement of needing 300 million photos put ViT out of reach for almost everyone. The community responded quickly with smarter, more accessible variants:

1. DeiT (Data-efficient Image Transformers): developed by Meta, it used distillation (a CNN «teacher» model that teaches the tricks to a Transformer «student») to prove that ViT could be trained competitively with ImageNet alone, without titanic datasets.
2. Swin Transformer: introduced hierarchical shifted window attention. Instead of looking at all 196 patches at once, it processes local windows that gradually merge and widen in the deeper layers. It’s the perfect fusion between a CNN’s local efficiency and a Transformer’s global flexibility, and today it’s one of the most widely used models in production.
3. Giant models (ViT-H, ViT-G, ViT-22B): colossal scales with billions of parameters that serve as the backbone for frontier vision engines like PaLI, Gemini, and Flamingo.

---

7. The definitive bridge to multimodality: CLIP

Here is ViT’s real legacy, and it’s not classifying photos 2% better than a CNN: it was unifying the mathematical formats of artificial intelligence.

Before ViT, language was processed as sequences (Transformers) and vision as continuous matrices (CNN). They were two worlds speaking incompatible languages, unable to understand each other.

By converting the image into a sequence of tokens, image and text became, structurally, the same thing.

The best example is CLIP (OpenAI):

• A ViT processes the image and returns a vector.
• A Transformer processes the text and returns another vector.
• Both models are trained together so that the cat photo and the phrase «a cat resting» land at the same coordinate point in a shared latent space.

From this common space came semantic image search (searching for photos by typing what you want, with no labels), zero-shot classification (recognizing objects the model never saw during training), generation tools like Stable Diffusion and Midjourney (which «understand» what to draw thanks to CLIP), and the eyes of models like GPT-4o Vision and Gemini Vision.

---

8. ViT vs. CNN today: who keeps the crown?

There’s no absolute winner. They’ve split the playing field according to hardware and task:

In short: CNNs still reign in the practical present (security cameras, phones, drones, portable medical devices), while ViT dominates the frontier (large foundation models, multimodal systems, and high-fidelity image generation).

The CNN dominates the present. The ViT dominates the future.

---

In summary

Vision Transformers marked an undeniable paradigm shift:

• They slice the image into 16×16 pixel patches and treat them as if they were words in a sentence.
• They add coordinates (Positional Encoding) so as not to lose the geometry, and a[CLS]token that summarizes everything for classification.
• They apply global attention from the first layer, letting any region of the photo relate directly to any other.
• They drop convolutions, kernels, and pooling, replacing them with pure, scalable attention layers.
• They need a lot of data to get going, but in exchange they scale with no ceiling.
• They enabled the multimodal revolution by unifying image and text under a single mathematical standard, serving as the foundation of CLIP, PaLI, Flamingo, Gemini Vision, and GPT-4o Vision.

It was the exact moment when images stopped being grids of pixels examined through a keyhole… and became language.