Dimensionality Reduction: PCA, t-SNE, UMAP

The Problem: Too Many Numbers to Draw

Open any cricket analytics page for an IPL player and you will find a wall of numbers: batting average, strike rate, boundary percentage, dot-ball percentage, economy rate, catches, run-outs, matches played, and a dozen more. A data analyst comparing thirty players is really comparing thirty points, each described by thirty numbers. A human being can draw a graph with two axes on paper, or three axes if we get clever with depth. Nobody can draw a thirty-axis graph. Yet the analyst still wants to answer very visual questions: which players are similar to each other? Are there natural groups — power hitters, anchors, all-rounders? Is there one player who stands apart from everyone else?

This is the exact problem dimensionality reduction solves. Every row of data — a player, a student, a photograph, a gene sample — lives in a space with as many dimensions as it has numeric features. A photograph that is 28 pixels wide and 28 pixels tall has 784 dimensions, one per pixel. A student's report card with ten subjects lives in a 10-dimensional space. Dimensionality reduction takes a point from a high-dimensional space and finds it an honest new address in a low-dimensional space — usually 2D or 3D — while keeping the parts of the data's structure that actually matter. "Honest" is the key word: a bad reduction just throws information away; a good one keeps the relationships between points intact even after most of the numbers are gone. This chapter covers three of the most important techniques for doing this: PCA, which is over a century old and still the first thing every data scientist reaches for; and t-SNE and UMAP, two much newer methods built specifically for visualizing data whose structure is curved rather than straight.

A Worked Example: Squeezing Two Subjects into One Score

Before any formulas, let's do this by hand with the smallest example that still shows the real idea: two subjects instead of thirty. Here are six students' marks (out of 100) in Mathematics and Science:

• A: Math 90, Science 85
• B: Math 80, Science 75
• C: Math 70, Science 65
• D: Math 50, Science 55
• E: Math 40, Science 45
• F: Math 30, Science 35

Notice these two subjects are strongly correlated — students who score high in Math also score high in Science. That correlation is exactly what dimensionality reduction exploits. If Math told us nothing about Science, we couldn't compress the two into one number without losing real information. But when they move together, a single combined score can stand in for both.

The average of all six Math marks is 60, and the average Science mark is also 60. To see how spread out the data is, we first "center" it by subtracting these averages from every point. This shifts the whole cloud of points so it's balanced around the origin (0, 0), which makes the arithmetic that follows much cleaner:

• A: (30, 25)
• B: (20, 15)
• C: (10, 5)
• D: (−10, −5)
• E: (−20, −15)
• F: (−30, −25)

Now ask a very concrete question: if we had to collapse each student down to a single number, which direction should we measure along, so that the six numbers we get are as spread out (as informative) as possible? A number that barely changes from student to student is useless for telling students apart; a number with a lot of spread preserves the differences between them.

Let's test three candidate directions by projecting (measuring the shadow of) each point onto that direction and computing the variance of the six resulting numbers.

Direction 1 — just use the Math score (ignore Science entirely). The six centered values are 30, 20, 10, −10, −20, −30. Squaring and averaging: (900+400+100+100+400+900)/6 = 466.7. That's the variance we capture — and it means we've thrown away all of the Science information.

Direction 2 — just use the Science score. Values: 25, 15, 5, −5, −15, −25. Variance: (625+225+25+25+225+625)/6 = 291.7. Worse than Math alone, because Science happens to vary a little less across these six students.

Direction 3 — a 45° diagonal, i.e., an equal blend of Math and Science: for each student, compute (Math\_centered + Science\_centered) / √2. For A that's (30+25)/1.414 ≈ 38.9; doing this for all six and squaring-and-averaging gives a variance of about 746.1. That is far higher than either single subject alone — because Math and Science move together, blending them lines the students up along a direction where the spread from both subjects adds up instead of being wasted.

This is the entire idea of Principal Component Analysis in one sentence: find the direction through the data that maximizes the variance of the projected points, use that as your new axis, and repeat for whatever variance is left over, always at right angles to the directions already chosen. The first such direction is called the first principal component (PC1); the second, perpendicular to it, is PC2; and so on. Testing a 45° line by hand was a lucky guess — in general the best direction is found by solving a small system of equations from the data's covariance matrix (a table that records how much each pair of features varies together), a calculation a computer performs instantly. For this dataset, solving it exactly gives a best direction tilted about 38.3° from the Math axis — slightly closer to Math than a plain 45° diagonal, because Math had the larger spread to begin with and PCA leans toward whichever original feature carries more variance. Along that exact direction, the variance captured is 756.1 — a little more than our 45° guess found.

Here is what that looks like geometrically. The blue dots are the six students plotted by their actual Math and Science marks. The amber line is PC1, the best-fit direction. The short dashed segments show each point's "shadow" (its perpendicular projection) onto that line — notice how short these dashed segments are, which is exactly what "the data lies close to this line" looks like. The teal strip below collapses everyone onto that single line, turning two marks into one combined score per student.

Two things are worth noticing in this picture. First, the ranking of the six PC1 scores (F lowest, A highest) matches the ranking you'd get from just averaging Math and Science — which makes sense, since PC1 turned out close to a 50-50 blend. Second, and more importantly, the total variance never disappeared: it just got reorganized. Var(Math) + Var(Science) = 466.7 + 291.7 = 758.3 exactly equals the sum of the variance captured by PC1 and the leftover variance along PC2 (756.1 + 2.2 = 758.3). PCA doesn't create or destroy information; it rotates the axes so that as much of the original spread as possible lands on the first few new axes, letting you discard the rest with minimal loss. Here, dropping PC2 loses only 2.2 out of 758.3 units of variance — about 0.3% — which is why compressing two marks into one combined score is safe for this class.

Formalizing It: What PCA Actually Does

Written as an algorithm, PCA on a dataset with n numeric features does four things:

1. Center the data: subtract each feature's mean, so the cloud of points is balanced around the origin.
2. Compute the covariance matrix: an n×n table where entry (i, j) records how feature i and feature j vary together (their covariance). The diagonal entries are just each feature's own variance.
3. Find the eigenvectors and eigenvalues of that matrix. Each eigenvector is a direction in the original n-dimensional space; its paired eigenvalue is exactly the variance you'd capture by projecting onto that direction. Sorting eigenvectors by eigenvalue, largest first, gives you PC1, PC2, PC3, and so on — each one guaranteed to be perpendicular to all the others.
4. Project the data onto however many of the top eigenvectors you want to keep. Keeping k of the original n dimensions gives you your reduced dataset.

You will never be asked to compute eigenvectors by hand for anything bigger than our 2×2 example — that's exactly the kind of repetitive linear algebra computers were built for. Here is that same six-student example run through the real library data scientists use:

import numpy as np
from sklearn.decomposition import PCA

marks = np.array([
    [90, 85],  # Math, Science
    [80, 75],
    [70, 65],
    [50, 55],
    [40, 45],
    [30, 35],
])

pca = PCA(n_components=1)
scores = pca.fit_transform(marks)

print(pca.explained_variance_ratio_)  # ~ [0.997]
print(pca.components_)                # ~ [[0.785, 0.620]]
print(scores.ravel())                 # ~ [39.0, 25.0, 10.9, -10.9, -25.0, -39.0]

Run this and you'll get numbers matching the hand calculation above: PC1 captures about 99.7% of the variance, its direction is close to (0.785, 0.620) — leaning toward Math, as we predicted — and the six scores line up exactly as the diagram showed. One quirk worth knowing: the sign of components_ is arbitrary. Some versions or runs might report (−0.785, −0.620) instead, flipping every score's sign. The magnitudes and the relative ordering of students never change; only which end of the line counts as "positive" does.

A Common Misconception

It's tempting to think dimensionality reduction just means "keep the most important original columns and drop the rest" — for example, keeping Math and throwing away Science because Math had the bigger variance. That is a different technique called feature selection, and PCA is not doing that. Notice that PC1's direction, (0.785, 0.620), uses both Math and Science — it is a new, manufactured feature, a weighted blend of the originals, not a copy of one of them. If you had thrown Science away entirely and kept raw Math scores, you'd have captured only 466.7 units of variance (61.6% of the 758.3 total). PCA's blended direction captures 756.1 units (99.7%) because it uses information from every original column at once. This is the single most important thing to internalize about PCA: the new axes are combinations of all the old ones, not a subset of them.

Where PCA Breaks Down: Curved Data

PCA's entire strategy rests on straight lines: it looks for a straight direction through the data along which variance is maximized. That works beautifully when the true structure in the data really is roughly flat, as it was for our correlated marks. But plenty of real data has structure that curves.

Picture a rolled-up dosa: a flat, roughly rectangular sheet, curled into a spiral so that its two ends are close together in ordinary 3D space, even though if you unrolled it, the point at the start of the roll and the point three full turns later are actually very far apart along the sheet. Now imagine you only get to measure straight-line (Euclidean) distance in 3D between points on that rolled-up sheet, and you're asked to flatten it back to 2D. A method like PCA, which cares only about maximizing straight-line variance, will happily slice a flat plane straight through the roll. Points from completely different layers of the roll — which were far apart along the sheet's actual surface — can end up looking close together in the 2D result, because they happened to be near each other in straight-line 3D distance. The reduction would be technically "high variance" but structurally wrong: it destroys the very neighborhood relationships that made the data meaningful.

This exact failure mode shows up in real datasets: images of a rotating object (angle of rotation is a curved, cyclic structure), single-cell gene-expression measurements (cells differentiate along branching, curved developmental paths), and word-embedding spaces. When the interesting structure in your data is a curved surface (called a manifold) sitting inside a high-dimensional space, you need a method that respects local neighborhoods rather than global straight-line distances. That's exactly what t-SNE and UMAP were built for.

t-SNE: Preserving Who's Next to Whom

t-SNE (t-distributed Stochastic Neighbor Embedding) abandons PCA's variance-maximizing approach completely and asks a much more local question for every single point: who are your close neighbors, and how close are they, relative to each other?

In the original high-dimensional space, t-SNE converts distances into probabilities: for each point, its nearby points get a high probability of being picked as its "neighbor," and far-away points get a probability close to zero. How many neighbors count as "nearby" is controlled by a parameter called perplexity, typically set between 5 and 50 — think of it as roughly "how many close friends should each point pay attention to." Then t-SNE places every point somewhere in a 2D or 3D layout and repeatedly nudges the points — using gradient descent, the same optimization idea used to train neural networks — so that the neighbor probabilities in the new low-dimensional layout match the neighbor probabilities from the original high-dimensional space as closely as possible. (The low-dimensional probabilities use a heavier-tailed distribution — the "t" in t-SNE — which gives points more room to spread apart in 2D than they had in high dimensions, fixing a tendency for everything to get crushed into the center that earlier neighbor-embedding methods suffered from.)

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
embedding = tsne.fit_transform(high_dimensional_data)

t-SNE is extremely good at revealing clusters that PCA would blur together — it's the standard way researchers visualize collections of handwritten digit images (grouping all the "7"s near each other, away from the "2"s) or single-cell genomics data (grouping cells of the same type). But it comes with a genuine misconception you need to guard against: the distance between two clusters in a t-SNE plot, and the size of a cluster, are not reliable measurements. t-SNE only promises to preserve who is near whom locally; it makes no promise about global distances or relative sizes. A cluster that looks small and tight might represent points that were actually quite spread out in the original space, and two clusters drawn far apart on the page are not necessarily more different from each other than two clusters drawn close together. Two separate runs of t-SNE on the same data, or the same run with a different perplexity, can also produce visibly different-looking layouts, because the optimization starts from random positions. Read a t-SNE plot for "which points cluster with which" — never for "how far apart are these two groups, really."

UMAP: A Faster, More Structure-Aware Cousin

UMAP (Uniform Manifold Approximation and Projection) shares t-SNE's core goal — preserve local neighborhoods when flattening curved, high-dimensional data — but reaches it differently. Instead of converting distances directly into Gaussian probabilities, UMAP first builds a graph connecting each point to its nearest neighbors (controlled by a parameter called n_neighbors, playing a role similar to perplexity), then uses ideas from topology — the branch of mathematics that studies how shapes stay connected when stretched or bent without tearing — to construct a low-dimensional layout whose neighbor-graph matches the original as closely as possible.

import umap

reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=42)
embedding = reducer.fit_transform(high_dimensional_data)

In practice, three differences from t-SNE matter most. First, speed: UMAP's optimization scales much better to large datasets, often finishing in a fraction of the time t-SNE takes on the same data. Second, UMAP tends to preserve more global structure — the relative positioning of clusters is somewhat more trustworthy than in t-SNE, though still not a precise distance measurement, so the same caution about reading too much into inter-cluster distances still applies. Third, and practically important, UMAP supports transforming brand-new points into an already-computed layout via a proper transform() step, something t-SNE does not support cleanly — t-SNE has to be re-run on the whole dataset (old points plus new) to place new points at all. This makes UMAP more usable as a real preprocessing stage in a production pipeline, not just a one-off visualization.

PCA vs t-SNE vs UMAP: When to Use Which

• PCA is linear, deterministic (same input always gives the same output, no sign flips aside), extremely fast even on huge datasets, and tells you exactly how much variance each component captures. Use it as a first look at any dataset, as a compression step before feeding data into another algorithm, or whenever the underlying structure is genuinely close to flat, as in our correlated-marks example.
• t-SNE is nonlinear, stochastic (results vary run to run and depend on the perplexity you choose), and computationally expensive on large datasets, but unmatched at revealing tight local clusters for a human to inspect visually. Use it purely for exploration and visualization — never trust cluster sizes or the distances between clusters, and never use it as a preprocessing step feeding into a downstream model.
• UMAP is nonlinear like t-SNE, but faster and better able to place new points after the fact. It has become the default choice for visualizing large, curved, high-dimensional datasets in fields like genomics and computer vision, and is often reasonable to use as an actual preprocessing step before clustering, unlike t-SNE.

Notice the general rule underneath all three: PCA answers "along which straight direction is my data most spread out?" while t-SNE and UMAP both answer a fundamentally different question, "for each point, which other points are its true neighbors, and can I recreate that same neighborhood pattern using far fewer dimensions?" Choosing between them is really choosing which question matches what you actually want to know about your data.

Check Your Understanding

1. A dataset has two features with variance 800 and 200, and a covariance between them of 0. What is PC1's direction, and how much variance does it capture? (Hint: covariance of 0 means the features don't move together at all — test the two axis directions as we did above.)
2. Why can't PCA's variance-maximizing strategy correctly "unroll" the rolled-dosa-shaped data described in this chapter, even though it is mathematically guaranteed to find the direction of maximum variance?
3. A classmate says, "In my t-SNE plot, Cluster X is drawn twice as far from Cluster Y as it is from Cluster Z, so Cluster X must be twice as similar to Cluster Z as to Cluster Y." What is wrong with this reasoning, and which of the three techniques in this chapter would give a more trustworthy answer to the underlying question?
4. You need to reduce a dataset of 50,000 sensor readings, each with 300 features, and later add newly arriving readings to the same 2D map without recomputing everything. Which technique fits this requirement, and why do the other two not?

Answers: (1) Since covariance is 0, there is no diagonal blend to gain from — the two features already are the principal directions. PC1 is simply the axis with variance 800 (the higher one), capturing 800/(800+200) = 80% of the total variance; PC2 is the other axis. (2) Maximum variance is a global, straight-line criterion, but the rolled sheet's meaningful structure is local and curved: two points that are close in straight-line 3D distance can belong to different layers of the roll and be very far apart along the sheet's true surface, so a flat "highest-variance slice" mixes unrelated layers together instead of unrolling them. (3) t-SNE plots don't preserve global inter-cluster distances, only local neighbor relationships, so the "twice as far means twice as different" reading is unsupported — the picture only licenses "points within a cluster are genuinely close," not comparisons between clusters. UMAP is somewhat more trustworthy on relative global placement, though still not a precise distance measurement, and PCA's explained_variance_ratio_ combined with actual projected distances would give the only rigorous numeric answer. (4) UMAP, because of its transform() method for placing new points into an existing embedding without recomputing the whole layout — PCA could technically project new points too (multiply by the same eigenvectors), but it would badly misrepresent curved sensor-data structure the way the rolled-dosa example showed; t-SNE has no clean way to add new points at all without rerunning on the full combined dataset.

Summary

Dimensionality reduction takes data described by many numbers and finds it a smaller, honest set of coordinates that preserves the structure that matters. PCA does this by rotating the axes to align with directions of maximum variance, using linear algebra (eigenvectors of the covariance matrix) to guarantee the first few new axes capture as much of the original spread as possible — our worked example showed two correlated subject marks compressing into a single score that kept 99.7% of the original variance, using a direction that blends both original features rather than discarding either one. PCA is fast, exact, and reversible, but only works well when the true structure is close to flat; curved, manifold-shaped data defeats its straight-line assumption. t-SNE and UMAP instead preserve local neighborhoods — who is close to whom — letting them faithfully unroll curved structure that PCA would tear apart, at the cost of losing PCA's exactness: t-SNE's layouts are stochastic and unreliable for reading global distances or cluster sizes, while UMAP trades a little of that local fidelity for speed and the ability to place new points into an existing map. Knowing which of these three questions you're actually asking about your data — "what's the best flat summary?" versus "what genuinely neighbors what?" — is what determines which tool belongs in your hands.