Dimensionality Reduction Methods Beyond PCA

When the Direction of Maximum Variance Is the Wrong Direction

Principal Component Analysis finds the direction along which your data spreads out the most, and projects the data onto it. That is a genuinely useful idea when your only goal is to keep as much of the raw information as possible while dropping dimensions. But PCA has a blind spot, and it is worth staring at directly before learning what replaces it. PCA never looks at labels. It does not know or care whether a point belongs to class A or class B; it only asks "where is the data spread out the most?" That question and the question "which direction best tells the two classes apart?" can have completely different answers, and when they do, PCA quietly throws away the exact information you needed.

Here is a case where this actually happens, not as a hypothetical but as an exact, checkable example. Suppose you have two clusters of points, each internally stretched out along the same diagonal direction, but the clusters themselves sit offset from each other along the direction perpendicular to that diagonal. This is a completely ordinary shape for real data to take — think of two exam-score cohorts where "hours studied" and "practice tests attempted" are positively correlated within each cohort, but one cohort (say, students who also took a coaching class) sits systematically to one side.

The diagram below plots exactly such a pair of clusters. Class A (blue) and Class B (red) are each spread along the same upward-sloping line, so the direction of maximum total variance in the combined data runs along that diagonal — this is exactly the direction PCA's first principal component would find. But projecting the data onto that diagonal makes every blue point line up exactly on top of a red point: the classes become completely indistinguishable. The direction perpendicular to it, which PCA would treat as low-variance and discard, is precisely the direction that separates the two classes perfectly.

This is not a rigged coincidence — it is the generic situation whenever the "spread within a class" and "distance between classes" point in different directions, which happens constantly with real, correlated features. The rest of this chapter is about the family of techniques that go beyond "just find the high-variance directions": methods that either use class labels to find directions that separate categories (Linear Discriminant Analysis), or abandon straight-line projections altogether to handle data that bends through space (t-SNE and autoencoders), or trade optimality for raw speed at massive scale (random projection).

Fisher's Linear Discriminant: A Direction That Uses the Labels

Linear Discriminant Analysis (LDA) solves exactly the problem the diagram above raises: given labelled data from two classes, find the one direction to project onto that keeps the classes as separated as possible. Unlike PCA, LDA is a supervised technique — it needs the class labels during training, and the direction it produces depends on those labels, not just on the raw spread of the points.

To make "separated as possible" precise, R. A. Fisher's 1936 criterion balances two things at once. Let the two classes have mean vectors μ₁ and μ₂. We want the projected class means, wᵀμ₁ and wᵀμ₂, to be as far apart as possible — that pushes the classes apart. But we also want each class to stay tightly clustered after projection — otherwise a large between-class gap is useless if each class is smeared wide enough to fill it. Fisher captured both goals in one ratio:

J(w) = (wT SB w) / (wT SW w)

Here S_B, the between-class scatter, is (μ₁ − μ₂)(μ₁ − μ₂)ᵀ — a matrix built purely from how far apart the two class means are. And S_W, the within-class scatter, is S₁ + S₂, where each S_i = Σ (x − μ_i)(x − μ_i)ᵀ summed over every point x in class i — this measures how spread out each class is around its own mean. J(w) is large exactly when the numerator (between-class spread after projecting onto w) is large relative to the denominator (within-class spread after projecting onto w). Maximizing J(w) is maximizing separation relative to noise, which is a far more useful target than maximizing raw variance.

Finding the w that maximizes J(w) is a calculus problem you can actually carry out. Differentiating J with respect to w and setting the result to zero (using the quotient rule, since J is a ratio) leads to the condition

SB w = J(w) SW w

which is a generalized eigenvalue problem. It looks intimidating, but S_B w has a special structure that collapses everything: since S_B = (μ₁−μ₂)(μ₁−μ₂)ᵀ, we get S_B w = (μ₁−μ₂) · [(μ₁−μ₂)ᵀw], and the bracketed term is just a number (a scalar), not a vector. So S_B w always points in the same direction as (μ₁−μ₂), regardless of w. Substituting this back and clearing the scalar factors, the optimal direction simplifies to:

w = SW-1 (mu1 - mu2)

In words: take the vector connecting the two class means, and "correct" it using the inverse of the within-class scatter. Where the classes are tightly clustered (small S_W in some direction), that correction stretches w further along that direction; where a class is loosely scattered, the correction shrinks it. This is exactly why LDA can find the discriminant direction in the diagram above even though it's the low-variance direction — S_W⁻¹ actively down-weights the high-variance (but useless) diagonal spread, and up-weights the low-variance (but decisive) perpendicular gap.

A fully worked example

Take two small classes in two dimensions — think of them as, say, five UPI transactions each, plotted by two normalized features (transaction frequency score, amount-variability score):

Class 1: (4,1), (2,4), (2,3), (3,6), (4,4)
Class 2: (9,10), (6,8), (9,5), (8,7), (10,8)

Step 1 — class means. Averaging each coordinate: μ₁ = (3, 3.6) and μ₂ = (8.4, 7.6).

Step 2 — within-class scatter per class. For Class 1, subtract μ₁ from every point, form the 2×2 outer product (x−μ₁)(x−μ₁)ᵀ for each point, and sum all five:

S1 = [ 4.0  -2.0 ]
     [-2.0  13.2 ]

Do the same for Class 2 around μ₂ = (8.4, 7.6):

S2 = [ 9.2  -0.2 ]
     [-0.2  13.2 ]

Step 3 — total within-class scatter. Add them:

SW = S1 + S2 = [ 13.2  -2.2 ]
                [ -2.2  26.4 ]

Step 4 — invert S_W. For a 2×2 matrix [[a,b],[c,d]], the inverse is (1/(ad−bc)) · [[d,−b],[−c,a]]. Here a=13.2, b=−2.2, c=−2.2, d=26.4, so the determinant is ad − bc = (13.2)(26.4) − (−2.2)(−2.2) = 348.48 − 4.84 = 343.64. Then:

SW-1 = (1/343.64) [ 26.4   2.2 ]  = [ 0.0768  0.0064 ]
                    [  2.2  13.2 ]    [ 0.0064  0.0384 ]

Step 5 — the discriminant direction. μ₁ − μ₂ = (3 − 8.4, 3.6 − 7.6) = (−5.4, −4.0). Multiplying:

w = SW-1 (mu1 - mu2)
w1 = 0.0768(-5.4) + 0.0064(-4.0) = -0.440
w2 = 0.0064(-5.4) + 0.0384(-4.0) = -0.188

w = (-0.440, -0.188)

Step 6 — check it worked. Project every point onto w by computing x·w for each: Class 1's five projected values fall between about −2.51 and −1.45, while Class 2's five projected values fall between about −5.91 and −4.15. The two intervals do not overlap at all — every single Class 1 point projects to a larger value than every single Class 2 point. That is a completely clean split from a single number per point, which is exactly what a good discriminant direction should deliver.

In code, you would never grind through these six steps by hand for real data — scikit-learn does it in one line:

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis(n_components=1)
X_projected = lda.fit_transform(X_train, y_train)

but doing it by hand once, as you just did, is what makes the geometry in the diagram earlier make sense instead of feeling like a magic formula.

Correcting a common misconception

A mistake students often carry over from PCA is assuming "the direction where the data spreads out most is automatically the direction that matters most." The worked example and the diagram both refute this directly: in the diagram, the max-variance direction gave zero separation (every blue point coincided with a red point), while a lower-variance direction gave perfect separation. Variance measures how spread out the data is, full stop — it says nothing about whether that spread is spread within a class (noise you want to average away) or spread between classes (signal you want to keep). PCA cannot tell these apart because it never sees the labels; LDA can, because S_W and S_B are built specifically to separate exactly those two kinds of spread.

A second thing worth being precise about: LDA for two classes gives you exactly one useful discriminant direction, because S_B = (μ₁−μ₂)(μ₁−μ₂)ᵀ has rank 1 (it's built from a single difference vector). With C classes, S_B has rank at most C−1, so LDA can extract at most C−1 discriminant directions — for a 10-class digit-recognition problem, that means at most 9 LDA axes, regardless of how many original pixel features you started with.

Curved Data and the Limits of Any Straight Line

Both PCA and LDA share a hidden assumption: they only ever consider straight-line (linear) projections — every output coordinate is a weighted sum of the input coordinates. This is fine when the useful structure in your data really does lie along flat directions. It breaks down when the data lies on a surface that curves through space.

Picture a jalebi — a strand of batter piped into a tight spiral before frying. The batter itself is genuinely one-dimensional: if you could walk along it, one number (distance travelled from the start) would tell you exactly where you are. But laid out in a plate, that one-dimensional strand occupies two full dimensions of physical space, curling back on itself. If you tried to describe position using ordinary straight-line coordinates (x, y) and then asked PCA to find "the important direction," PCA would find some diagonal that captures a lot of variance, but a point near the start of the spiral and a point several loops later could project to nearly the same location on that line, even though they are far apart along the actual strand. A linear projection cannot "unroll" a spiral, because unrolling is fundamentally a bending operation, not a projection along a fixed direction.

Real datasets exhibit this constantly — this is called the manifold hypothesis: high-dimensional data (images, sensor readings, word embeddings) often lies near some lower-dimensional curved surface embedded in the high-dimensional space, rather than along a flat subspace. Handwritten digit images, for instance, occupy a tiny curved region of pixel-space compared to the space of all possible pixel arrangements — as a digit's stroke width or slant changes smoothly, the corresponding image traces out a smooth, curved path through pixel-space, not a straight line. This is exactly the setting where PCA and LDA — both fundamentally linear — hit a wall, and where the next two methods, t-SNE and autoencoders, are designed to work.

t-SNE: Preserving Who Is a Neighbour, Not the Actual Distance

t-distributed Stochastic Neighbour Embedding (t-SNE) takes a completely different strategy from PCA and LDA. Instead of finding a direction to project onto, it asks a local question at every point: "which other points are my close neighbours?" — and then tries to arrange points in two or three dimensions so that the neighbour relationships survive, even if the actual distances get badly distorted.

Concretely, for every pair of points i and j in the original high-dimensional data, t-SNE converts the distance between them into a probability that j would be picked as a neighbour of i, using a Gaussian centred on i:

p(j|i) = exp(-||xi - xj||^2 / (2 * sigma_i^2)) / sum over k!=i of exp(-||xi - xk||^2 / (2 * sigma_i^2))

Nearby points get high probability, far points get probability close to zero — the Gaussian's width σ_i is chosen separately for each point i so that a fixed "effective number of neighbours," called the perplexity (typically set between 5 and 50), is captured. In a dense region of the data, σ_i comes out small; in a sparse region, σ_i comes out larger, so every point gets a comparably sized neighbourhood regardless of local density. These per-point probabilities are then symmetrized into a single p_ij = (p(j|i) + p(i|j)) / (2n) for a dataset of n points.

Now t-SNE places every point somewhere in the low-dimensional map, at position y_i, and defines a similar probability there — but using a Student-t distribution with one degree of freedom instead of a Gaussian:

q_ij = (1 + ||yi - yj||^2)^-1 / sum over k!=l of (1 + ||yk - yl||^2)^-1

The heavy tail of the t-distribution (compared to a Gaussian) is not a minor technical detail — it is what makes the whole method work. In high dimensions there is a lot of "room": a point can have many other points at roughly the same, moderate distance from it. Two or three dimensions have far less room, so if the low-dimensional map tried to preserve those moderate distances using a thin-tailed Gaussian, all the points would be forced to crowd unnaturally close together (this is called the crowding problem). The t-distribution's heavy tail lets moderately-distant points in the map end up quite far apart while still contributing a reasonably large q_ij, relieving the crowding and letting distinct clusters visibly separate.

Training then becomes an optimization: move every y_i to make the low-dimensional distribution Q match the high-dimensional distribution P as closely as possible, measured by Kullback–Leibler divergence, KL(P‖Q) = Σᵢⱼ p_ij log(p_ij/q_ij), minimized by gradient descent. Points that were close in the original space are pulled together in the map; points that were far apart are pushed apart.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_2d = tsne.fit_transform(X_high_dim)

If you ran this on, say, vectors of IPL batting statistics (strike rate, average, boundary percentage, dot-ball percentage, and so on) across many players, players with genuinely similar batting styles would tend to land near each other in the resulting 2-D map, even though "similar style" is not any single one of the original stats — it is a pattern spread across all of them that t-SNE's neighbour-matching can pick up on.

A misconception to correct explicitly: it is extremely tempting to look at a t-SNE plot and read meaning into cluster sizes and the distances between clusters — "cluster X is much bigger than cluster Y" or "cluster A is closer to cluster B than to cluster C." Both readings are unreliable. Because t-SNE only tries to preserve local neighbour probabilities, not global distances, the optimization is free to stretch some regions of the map and compress others as long as nearby points stay nearby — apparent cluster size in the 2-D output does not correspond to how spread out that group actually is in the original space, and the gap between two well-separated clusters carries no calibrated meaning at all. The only thing you can trust in a t-SNE plot is which points are near each other; treat everything else as visual artifact of the optimization, not as measured fact.

It is also worth being precise about where t-SNE's probability machinery does and does not connect to your board syllabus. The idea of converting distances into a probability distribution and comparing two distributions is, at its core, an application of basic probability reasoning — assigning a likelihood to each possible neighbour, which is conceptually continuous with the Class 11–12 probability you already know. But the specific tool used here — a continuous Gaussian probability density with a variance parameter σ² — is not part of the core CBSE Class 12 Mathematics Probability chapter, which focuses on conditional probability, Bayes' theorem, and discrete random variables; the Gaussian/normal density with its bell-curve formula is treated properly in the Applied Mathematics elective (and in Statistics for Economics), so if you have taken core Maths only, treat σ_i here as "a knob controlling neighbourhood width" rather than expecting to have seen its formal definition already.

Autoencoders: Let a Neural Network Learn the Compression

Autoencoders take yet another approach: instead of a fixed mathematical formula for the projection, train a neural network to invent one. An autoencoder is built from two connected networks. The encoder, f, takes an input x (say, a 784-number vector, one number per pixel of a 28×28 handwritten digit image) and compresses it down to a much smaller vector z = f(x) — the bottleneck or latent representation, perhaps only 32 numbers. The decoder, g, takes that compressed vector and tries to reconstruct the original input, producing x̂ = g(z) = g(f(x)).

The network is trained purely to make the reconstruction accurate — minimizing a reconstruction loss such as mean squared error, ||x − x̂||², averaged over the whole training set. Notice there are no class labels anywhere in this loss; the network never needs to be told what digit an image shows. It is forced to discover, on its own, which 32 numbers are enough to reconstruct any typical handwritten digit reasonably well — and those 32 numbers necessarily capture the structure that matters (stroke shape, slant, thickness) while discarding pixel-level noise that doesn't help reconstruction.

from tensorflow import keras
from tensorflow.keras import layers

input_dim = 784
latent_dim = 32

encoder_input = keras.Input(shape=(input_dim,))
encoded = layers.Dense(128, activation='relu')(encoder_input)
encoded = layers.Dense(latent_dim, activation='relu')(encoded)

decoded = layers.Dense(128, activation='relu')(encoded)
decoded = layers.Dense(input_dim, activation='sigmoid')(decoded)

autoencoder = keras.Model(encoder_input, decoded)
encoder = keras.Model(encoder_input, encoded)

autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(X_train, X_train, epochs=20, batch_size=256)

compressed = encoder.predict(X_test)

Notice the training call: autoencoder.fit(X_train, X_train, ...) — the input and the target are the same array. This is why autoencoders are called self-supervised: they don't need separately collected labels, because the label is just the input itself.

Here is a genuinely satisfying theoretical fact that ties this whole chapter back to where it started. If every layer of the encoder and decoder used a purely linear activation (no ReLU, no sigmoid) and the loss is mean squared error, it can be proven that the optimal such autoencoder learns a latent space that spans exactly the same subspace as PCA's top principal components — a linear autoencoder with a k-dimensional bottleneck is, in a precise mathematical sense, computing PCA. The power of autoencoders over PCA only appears once you introduce nonlinear activations like ReLU or sigmoid in the hidden layers, as in the code above: a nonlinear autoencoder can bend its compression to follow a curved manifold — it can, in principle, learn to "unroll the jalebi" that a linear method like PCA or LDA cannot, because the encoder function is no longer restricted to straight-line projections.

Random Projection and the Johnson–Lindenstrauss Guarantee

Every method so far — PCA, LDA, t-SNE, autoencoders — spends real computational effort finding a "good" direction or mapping, tailored to the specific dataset in front of it. Random projection throws that idea out entirely: pick the projection directions completely at random, with no reference to the data at all, and use those.

This sounds like it should fail badly, and yet it provably does not, as long as you only care about preserving pairwise distances approximately. This is the content of the Johnson–Lindenstrauss (JL) lemma: given any n points in a high-dimensional space (dimension d, however large) and any small tolerance ε between 0 and 1, there exists a linear map down to a much lower dimension k — with k growing only like (log n)/ε², completely independent of the original dimension d — such that every pairwise distance among the n points is preserved up to a factor of (1 ± ε). Astonishingly, a random projection (for example, each entry of the projection matrix drawn independently from a Gaussian distribution, then the matrix rescaled) satisfies this guarantee with high probability — you don't need to search for a good projection at all; almost any random one works.

To get a feel for the numbers: one commonly used (conservative) form of the bound states k ≥ 8·ln(n)/ε². For n = 1,000 points and a tolerance of ε = 0.2 (allowing pairwise distances to shift by up to 20%), that works out to k ≥ 8 × ln(1000) / 0.04 ≈ 8 × 6.91 / 0.04 ≈ 1382. If your original data lived in d = 100,000 dimensions, collapsing it down to roughly k ≈ 1,382 random directions — a more than 70-fold reduction — is enough to keep every one of the ½ × 1000 × 999 pairwise distances within 20% of its original value, guaranteed with high probability, without ever looking at the data.

from sklearn.random_projection import GaussianRandomProjection

rp = GaussianRandomProjection(n_components=1382, random_state=0)
X_reduced = rp.fit_transform(X)

Why does this matter when LDA and autoencoders exist and typically produce better, more meaningful reduced representations? Speed and scale. Random projection needs no training loop, no gradient descent, no matrix inversion of a scatter matrix — you multiply by a fixed random matrix once, and you're done, in time roughly proportional to the size of the data. This matters enormously once n itself becomes huge. India's Aadhaar system, for instance, must check every new biometric enrollment against a database of well over a billion existing residents to prevent duplicates — at that scale, comparing high-dimensional feature vectors directly is expensive enough that a fast, data-independent dimensionality reduction as a cheap preprocessing step, before a more careful nearest-neighbour search, becomes genuinely attractive in a way that a slow, carefully-optimized method cannot match. Random projection is the tool you reach for when the dataset is too large to afford anything smarter, and the JL lemma is the guarantee that tells you random is still safe.

Choosing Among Them

• PCA — unsupervised, strictly linear, keeps the directions of maximum total variance. Use it as a default first step, or whenever you have no labels at all.
• LDA — supervised, strictly linear, keeps the direction(s) that best separate known classes; limited to at most C−1 output dimensions for C classes. Use it when you have labels and your real goal is classification, not just compression.
• t-SNE — unsupervised, nonlinear, preserves local neighbour structure for visualization in 2 or 3 dimensions; distances and cluster sizes in its output are not quantitatively meaningful, and it does not give you a reusable mapping for new points. Use it to visually explore whether structure exists in your data, never as a preprocessing step feeding into another model.
• Autoencoders — self-supervised, nonlinear (when using nonlinear activations), learns a reusable encoder function via gradient descent; needs a reasonably large training set and compute budget to train well. Use it when PCA's linear subspace is visibly too restrictive for your data (curved/manifold structure) and you have enough data and training time to learn a good encoder.
• Random projection — unsupervised, linear, data-independent, essentially free to compute, backed by the Johnson–Lindenstrauss guarantee on pairwise distances. Use it as a cheap first-pass reduction before a slower downstream algorithm, especially when the number of points n is enormous.

Test Yourself

1. Two classes have S_W = [[8, 0], [0, 2]] (already summed across both classes) and means μ₁ = (1, 5), μ₂ = (5, 1). Without inverting anything, argue from the structure of S_W alone whether the optimal LDA direction w will point closer to the horizontal axis or the vertical axis, and explain why using the idea of "correcting μ₁−μ₂ by S_W⁻¹."
2. Explain, using the crowding-problem idea, why t-SNE uses a heavy-tailed Student-t distribution for similarities in the low-dimensional map but a Gaussian for similarities in the original high-dimensional space — why can't it just use a Gaussian in both places?
3. A linear autoencoder with a 2-dimensional bottleneck is trained with mean squared error loss on a dataset. A classmate claims "this autoencoder can learn any 2-dimensional curved manifold the data lies on, same as a nonlinear one." Is this correct? Justify your answer using what determines whether an autoencoder's solution matches PCA.
4. Given Class A = {(0,0), (2,1)} and Class B = {(0,4), (2,6)}, compute the LDA direction w by hand, following the same six steps as the worked example: class means, per-class scatter, total S_W, det(S_W), S_W⁻¹ via the 2×2 inversion formula, then w = S_W⁻¹(μ_A − μ_B). (Check your work: S_W should come out as [[4,3],[3,2.5]] with determinant 1, giving a clean invertible matrix.)
5. A dataset has n = 10,000 points sitting in d = 50,000 dimensions. Using the conservative bound k ≥ 8·ln(n)/ε², compute the minimum k needed to preserve all pairwise distances within ε = 0.1 (10%) under a random projection. Compare this k to the original dimension d, and state in one sentence why the JL lemma's independence from d is what makes this useful.

Summary

PCA finds the directions of maximum variance without ever looking at labels, and that is both its strength (it needs no labelled data) and its weakness (it can and does destroy exactly the structure that separates classes, as the opening diagram showed concretely). Linear Discriminant Analysis fixes this by using labels directly: it maximizes Fisher's ratio J(w) = (wᵀS_Bw)/(wᵀS_Ww), and the maximizing direction works out to w = S_W⁻¹(μ₁−μ₂) — a mean-difference vector reshaped by the inverse of how spread out each class is internally. Both PCA and LDA are fundamentally linear, so both fail when the true structure in the data lies along a curved manifold rather than a flat subspace — the jalebi problem. t-SNE handles curved, complex structure for visualization by matching neighbour probabilities (Gaussian in the original space, heavy-tailed Student-t in the low-dimensional map, matched by minimizing KL divergence), at the cost of its output distances and cluster sizes carrying no reliable quantitative meaning. Autoencoders handle curved structure by training a neural network end-to-end to compress and reconstruct, with the elegant fact that a purely linear autoencoder collapses back to computing PCA, while nonlinear activations let it do genuinely more. And when the dataset is simply too enormous for any of these to run in reasonable time, random projection sacrifices data-awareness entirely, relying on the Johnson–Lindenstrauss lemma's guarantee that even a random linear map preserves pairwise distances approximately, using a target dimension that depends only on how many points you have and how much distortion you can tolerate — never on how many dimensions you started with.

Think About It

Think about this: How would you explain dimensionality reduction methods beyond pca to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.