Contrastive Learning: Learning from Unlabeled Data

Imagine you are building the photo-search feature for an app used by tourists at the Taj Mahal. Every day, thousands of visitors upload photos — the same white marble dome shot at sunrise, at sunset, in monsoon fog, cropped tight on the minarets, or taken from across the Yamuna river. Nobody sits down and labels each one "Taj Mahal, front view" or "Taj Mahal, side view." There are no labels at all. Yet you want the app to know that two very different-looking photos are pictures of the same monument, and that a third photo of the Red Fort is something else entirely. How can a machine learn this without ever being told the name of a single monument?

This is the exact problem that contrastive learning was invented to solve. It is a way of training a neural network to organize unlabeled data — photos, sentences, audio clips — into a meaningful map, purely by teaching it one simple relationship: which things belong together, and which things don't. No human ever writes "this is a Taj Mahal photo." The network figures out that structure entirely on its own, using nothing but the raw data.

The Problem With Needing Labels for Everything

The AI systems you have studied so far — a model that predicts house prices, or classifies an email as spam — are trained on labeled data. Every house price example comes with the actual price attached; every spam example comes tagged "spam" or "not spam." This works well, but it has an expensive catch: someone has to sit down and label every single example by hand. For a dataset of a few thousand emails, that's manageable. For the billions of photos uploaded to the internet every day, it is simply impossible — no company can afford to pay humans to look at and label a billion images.

At the same time, unlabeled data is everywhere and free — every photo ever uploaded, every sentence ever written, every audio clip ever recorded. The question that drives contrastive learning is: can we squeeze useful structure out of all that unlabeled data, without paying anyone to label it?

The Key Idea: Make the Data Label Itself

Here is the trick, and it is genuinely clever. Take one photo of the Taj Mahal. Now create two different-looking versions of that exact same photo — crop one tightly on the dome, rotate the other slightly, change its colours, flip it horizontally. These two altered images look somewhat different pixel-by-pixel, but they are unmistakably photos of the same physical object. Call this pair of images an anchor and a positive.

Now grab a completely unrelated photo from the dataset — say, a photo of Qutub Minar. Pair it with the anchor too. This pair is called a negative.

The network never sees the words "Taj Mahal" or "Qutub Minar." It only sees three images and one instruction, repeated over millions of examples: make the anchor and the positive end up close together, and make the anchor and the negative end up far apart. The "label" for each pair — same or different — was generated automatically from the fact that the positive was manufactured by altering the anchor. This is why the technique is called self-supervised learning: the supervision (the correct answer) comes from the structure of the data itself, not from a human labeller.

From Pixels to Points: What Is an Embedding?

To make "close together" and "far apart" precise, we need a way to turn an image into numbers. A neural network called an encoder takes an image and outputs a list of numbers — a vector — called an embedding. Real embeddings used in systems like SimCLR typically have 128, 512, or even 2048 numbers in them, one for every dimension the network has learned to care about (edges, textures, shapes, colours, and hundreds of more abstract patterns no human named). We obviously cannot draw a 512-dimensional space on paper, so throughout this chapter we will use a toy embedding with just 2 numbers — enough to plot on an ordinary x-y grid and compute by hand, while keeping exactly the same idea real systems use.

Once an image becomes a point in this space, "closeness" becomes an ordinary distance you already know how to calculate — the same distance formula from coordinate geometry:

distance(P, Q) = sqrt( (P_x - Q_x)^2 + (P_y - Q_y)^2 )

A small distance means two embeddings are close together (the network thinks they're similar); a large distance means they are far apart (the network thinks they're different). Note: production systems like SimCLR usually measure closeness with cosine similarity — comparing the angle between two vectors rather than the straight-line distance between them — because it ignores overall brightness/scale and focuses on direction. The pull-and-push intuition is identical either way; we'll stick to plain distance here because the arithmetic is easy to trace by hand.

Worked Example: Watch the Pull and the Push Happen

Let's build a tiny, fully traceable example. Suppose an encoder (before any training) has already converted four Indian-monument photos into 2-D embeddings:

• anchor — Taj Mahal, photo 1
• positive — Taj Mahal, photo 2 (a cropped, rotated version of photo 1 — the augmented "same object" pair)
• negative1 — a photo of Qutub Minar (a completely different monument)
• negative2 — a photo of Red Fort (also a completely different monument)

Here is the code that computes how far apart each pair sits in embedding space, before any contrastive training has happened:

import math

def distance(p, q):
    return math.sqrt((p[0] - q[0])**2 + (p[1] - q[1])**2)

# 2-D "embeddings" BEFORE training (toy example; real ones have hundreds of dimensions)
anchor    = (2.0, 3.0)   # Taj Mahal, photo 1
positive  = (5.0, 6.0)   # Taj Mahal, photo 2 (augmented view of photo 1)
negative1 = (2.5, 3.2)   # Qutub Minar (unrelated monument)
negative2 = (6.0, 5.0)   # Red Fort (unrelated monument)

print("distance(anchor, positive)  =", round(distance(anchor, positive), 3))
print("distance(anchor, negative1) =", round(distance(anchor, negative1), 3))
print("distance(anchor, negative2) =", round(distance(anchor, negative2), 3))

distance(anchor, positive)  = 4.243
distance(anchor, negative1) = 0.539
distance(anchor, negative2) = 4.472

Trace the arithmetic yourself for the first line: the anchor is at (2.0, 3.0) and the positive is at (5.0, 6.0), so the differences are (5.0 − 2.0) = 3.0 and (6.0 − 3.0) = 3.0. Squaring and adding gives 3.0² + 3.0² = 9 + 9 = 18, and √18 ≈ 4.243 — matching the printed output exactly.

Now look at what these three numbers are telling us, ranked from smallest to largest: negative1 (0.539) is closest to the anchor, then the positive (4.243), and finally negative2 (4.472) is farthest of all. This is a problem. Before training, the true positive — another photo of the very same Taj Mahal — sits farther from the anchor than an unrelated Qutub Minar photo does. That is exactly backwards from what a well-trained encoder should produce: an untrained network has no idea yet that two crops of the same building belong together, so it is fooled by superficial pixel similarity between the anchor and negative1's crop, colour, or angle. The genuinely matching pair (anchor and positive) isn't even the closest pair, and Qutub Minar being closer to the anchor than the Taj Mahal's own second photo is exactly the mistake contrastive training exists to fix.

Now compare that to what a well-trained encoder should produce. After many rounds of contrastive training, the same four photos might land like this:

# 2-D "embeddings" AFTER contrastive training (illustrative target state)
anchor     = (2.0, 3.0)
positive   = (2.3, 3.4)   # pulled in close to the anchor
negative1  = (8.0, 1.0)   # pushed far away
negative2  = (0.5, 9.0)   # pushed far away

print("distance(anchor, positive)  =", round(distance(anchor, positive), 3))
print("distance(anchor, negative1) =", round(distance(anchor, negative1), 3))

distance(anchor, positive)  = 0.5
distance(anchor, negative1) = 6.325

Now the two Taj Mahal photos sit only 0.5 units apart, while the unrelated Qutub Minar photo has been pushed out to 6.325 units — more than twelve times farther. Nothing about the raw pixels of Qutub Minar changed; what changed is that the encoder learned which visual features actually indicate "the same object" (structure, proportions, shape) versus which features are irrelevant noise (lighting, crop, exact angle). That shift in the network's weights is entirely the result of being shown millions of pull-together / push-apart examples like this one.

Turning "Pull and Push" Into a Number a Computer Can Optimize: The Contrastive Loss

Saying "pull the positive closer, push the negative away" is intuitive, but to actually train a neural network with gradient descent, we need a single number — a loss — that is large when the network is doing badly and small (ideally zero) when it's doing well. A simple version of this, called triplet loss (introduced for face-recognition embeddings by Schroff, Kalenichenko, and Philbin at Google in the 2015 FaceNet paper), does exactly this:

loss = max(0, distance(anchor, positive) - distance(anchor, negative) + margin)

Here margin is a small buffer we choose ahead of time (say, 1.0) — it says "I don't just want the negative farther than the positive, I want it farther by at least this much, with room to spare." Let's trace this formula through our before-and-after numbers, using negative1 (the closest, and therefore most troublesome, negative) with margin = 1.0:

def triplet_loss(d_pos, d_neg, margin=1.0):
    return max(0, d_pos - d_neg + margin)

# BEFORE training
print(round(triplet_loss(4.243, 0.539), 3))   # -> 4.704  (big loss: badly wrong)

# AFTER training
print(round(triplet_loss(0.5, 6.325), 3))     # -> 0.0    (zero loss: satisfied)

Before training: 4.243 − 0.539 + 1.0 = 4.704, a large positive number — the network is punished heavily, because its positive pair is far apart while its negative pair is suspiciously close. After training: 0.5 − 6.325 + 1.0 = −4.825, and since the formula takes max(0, ...), the loss floors out at exactly 0 — the network has more than satisfied the margin, so gradient descent stops pushing on this triplet and moves on to harder ones. This single number, computed for millions of anchor/positive/negative triplets and averaged, is what gradient descent actually minimizes during training — it converts our visual intuition of "pull and push" into something calculus can act on.

Modern large-scale systems like SimCLR (Chen et al., Google Research, 2020) use a more powerful cousin of this idea called the NT-Xent loss (built on a formula called InfoNCE), which compares the positive pair not against just one negative but against every other image in the training batch simultaneously, using a softmax. That lets the network learn from dozens of negatives at once instead of one at a time. The underlying intuition — minimize distance for true pairs, maximize it for everything else — is unchanged; only the bookkeeping got more efficient.

Seeing It on a Map

The diagram below plots the four before-training embeddings from our worked example on an ordinary 2-D grid, with dashed lines showing all three relationships radiating from the anchor.

Notice how visually cramped the anchor and negative1 are compared to the anchor and its true positive — that gap between "what the untrained network thinks" and "what is actually true" is precisely the signal that gradient descent uses to update the encoder's weights, pulling the blue line shorter and stretching both red lines longer with every training step.

A Common Misconception: "No Labels" Doesn't Mean "No Supervision"

A mistake students often make when they first hear "contrastive learning trains on unlabeled data" is to assume this means the network learns with no correct-answer signal at all — the way you might notice patterns just by staring at unsorted objects. That's wrong, and it's worth stating precisely why.

Contrastive learning does have a correct answer for every single training example — it is just generated automatically instead of typed in by a human. When we crop and rotate a Taj Mahal photo to create its positive pair, we — the algorithm designer — already know, with total certainty, that this pair is "same object." That fact came from how we constructed the pair, not from a human annotator looking at the image and writing "Taj Mahal." This is precisely why the field calls it self-supervised learning, not unsupervised learning: there is still a supervisory signal (the loss function has a definite right answer to push toward), but the signal was manufactured from the data's own structure — via augmentation — rather than sourced from a person. Unsupervised learning techniques you may encounter elsewhere, like k-means clustering, genuinely have no correct-answer signal to check against; contrastive learning is a different, more targeted category that sits between fully supervised and fully unsupervised learning.

How Real Systems Actually Build Positive Pairs

Our worked example hand-waved "crop, rotate, change colours" — but choosing the right set of transformations (called augmentations) turns out to matter enormously, and different data types need entirely different augmentation strategies:

• Images (used by SimCLR): random cropping, horizontal flipping, colour jittering (shifting brightness/contrast/saturation), and converting to grayscale. The idea is to change everything about the pixels that a human wouldn't consider relevant to "what object is this," while leaving the object's actual identity untouched.
• Image + text pairs (used by CLIP, released by OpenAI in 2021): instead of two augmented views of the same image, the positive pair is an image and its real caption scraped from the internet — for example, a photo of the Taj Mahal paired with the text "The Taj Mahal at sunrise, Agra." The negative pairs are that same image matched against captions belonging to completely different photos in the batch. This is how CLIP learns to connect pictures and language without anyone hand-labelling "this photo is a monument."
• Words (an early precursor): word2vec, introduced by Mikolov and colleagues at Google in 2013, used a related idea years before SimCLR or CLIP existed. It trained word embeddings by treating a word and its real neighbouring words in a sentence as a "positive" pair, and a word paired with a randomly sampled, unrelated word as a "negative" pair — a technique called negative sampling. It didn't use the phrase "contrastive learning," but the pull-real-neighbours-together, push-random-words-apart mechanism is the same core idea this chapter has been building toward.

Getting augmentations wrong breaks the whole method. If you cropped so aggressively that the crop no longer contained the Taj Mahal's dome at all, you'd be teaching the network that two completely different-looking things are "the same" — poisoning the very signal you're trying to create.

Why This Matters When Labels Are Scarce

The practical payoff of all this is transfer learning under label scarcity. Suppose an Indian startup is building handwriting recognition for a regional script — say, Gujarati or Malayalam — where scanned, hand-labelled training samples are expensive and scarce, but millions of unlabelled scanned pages exist in libraries and government archives. The standard workflow is: first, pre-train an encoder on the huge pile of unlabelled pages using contrastive learning, so it learns good general-purpose visual features (stroke shapes, curves, spacing) with zero labelling cost. Only afterward do you fine-tune that already-competent encoder on the small labelled dataset you can actually afford to build. The contrastive pre-training step does the heavy lifting; the scarce labelled data just needs to teach the final, much easier step of mapping learned features to specific character classes. This two-stage pattern — contrastive pre-train on abundant unlabelled data, then fine-tune on scarce labelled data — is the specific reason contrastive learning became central to modern AI rather than a laboratory curiosity.

Practice: Test Your Understanding

1. Trace it by hand. Two embeddings are a = (1.0, 1.0) and b = (4.0, 5.0). Compute distance(a, b) by hand, showing the squared differences before taking the square root.
2. Spot the setup. A model is given three embeddings from a triplet: anchor, positive, and negative, with distances distance(anchor, positive) = 1.2 and distance(anchor, negative) = 3.8. Using margin = 1.0, compute the triplet loss. Is the network doing well or badly on this triplet? Explain using the number you computed.
3. Explain the misconception. A classmate says, "Contrastive learning is unsupervised because there are no labels." Write two or three sentences correcting this, using the term "self-supervised" and explaining where the correct-answer signal actually comes from.
4. Design a positive pair. You are building a contrastive learning system for Indian classical dance photos (Bharatanatyam, Kathak, etc.), where the goal is for the encoder to recognize the same dance pose photographed from different camera angles. Suggest two augmentations you would apply to create a positive pair, and one augmentation you should avoid because it would destroy the information the network needs to learn (explain why it would hurt).
5. Compare systems. In one or two sentences each, state what forms the "positive pair" in (a) SimCLR and (b) CLIP.

Summary

• Contrastive learning trains an encoder using unlabeled data by constructing positive pairs (two views of the same underlying thing, usually via data augmentation) and negative pairs (unrelated examples), then optimizing the network to pull positive-pair embeddings close together and push negative-pair embeddings far apart.
• An embedding is a vector of numbers produced by an encoder network; "closeness" between embeddings is measured with a distance formula (or, in production systems, cosine similarity), directly extending the coordinate-geometry distance formula you already know.
• A triplet loss — max(0, distance(anchor, positive) − distance(anchor, negative) + margin) — turns the pull/push intuition into a single number gradient descent can minimize; modern systems like SimCLR use the more powerful NT-Xent/InfoNCE loss, which compares a positive pair against every other example in the batch at once.
• This is called self-supervised learning, not unsupervised learning, because the correct answer for every training pair is still known with certainty — it is generated automatically from how the pair was constructed, not typed in by a human labeller.
• SimCLR (Chen et al., Google Research, 2020) builds positive pairs from two augmented views of the same image; CLIP (OpenAI, 2021) builds them from an image and its real caption; word2vec (Mikolov et al., Google, 2013) applied the same pull-real-neighbours/push-random-words idea to text years earlier through negative sampling.
• The practical payoff: contrastive pre-training on cheap, abundant unlabeled data followed by fine-tuning on a small labelled dataset lets teams build strong models even when hand-labelled data — for a regional script, a niche domain, or a local dataset — is scarce and expensive.