PCA: Dimensionality Reduction Wizard

A CBSE school's annual assessment sheet for one section of Class 10 typically has more columns than you'd think: marks in Mathematics, Science, Social Science, English, a second language, plus attendance percentage, a co-scholastic grade, a sports score, and an arts score. That is already nine numbers per student. Add a few more enrichment scores and you can easily reach thirteen or fourteen. If the school wants to plot students on a single chart to spot patterns — who is broadly strong, who is broadly struggling, who is unusual — a chart needs two axes, maybe three. You cannot put fourteen numbers on a page and still see anything. Yet you don't want to just pick two subjects and throw the rest away, because a student's Science and Maths marks usually move together, and their Social Science and English marks usually move together too — the columns are not independent information, they are correlated echoes of a smaller number of underlying "themes" (something like overall academic strength, and something like language ability). Principal Component Analysis (PCA) is the technique that finds those underlying themes mathematically, and tells you exactly how much of the original spread in the data each one accounts for. That is the actual job of PCA: not "compress data" in some vague sense, but find the smallest number of new, uncorrelated axes that capture the largest possible share of the original variance.

The core idea, in two dimensions first

Before touching fourteen columns, understand what PCA does with two. Suppose you plot every student's Mathematics mark against their Physics mark. If the two subjects are correlated — which they usually are, since both reward similar problem-solving habits — the cloud of points on the scatter plot is not a circle. It is a tilted, cigar-shaped ellipse: students who score high in Maths tend to also score high in Physics, so the cloud stretches along a diagonal direction rather than spreading evenly in the horizontal and vertical directions.

Here is the key realization PCA is built on: the natural axes to describe this cloud are not the original "Maths axis" and "Physics axis" at all. The natural axes are the long direction of the ellipse (call it a new axis) and the short direction perpendicular to it (a second new axis). Along the long axis, the data varies a lot — this is where the real, informative differences between students live. Along the short axis, the data barely varies — this is closer to noise, or at least much less informative. If you had to keep only one number per student instead of two, keeping the coordinate along the long axis loses far less information than keeping either the original Maths mark or the original Physics mark alone.

PCA is the procedure that finds this long axis (called the first principal component, or PC1) and the perpendicular short axis (the second principal component, PC2) exactly, using linear algebra rather than eyeballing an ellipse. Once you have more than two original columns, the same idea just continues into more dimensions: PCA finds a new set of mutually perpendicular axes, ranked by how much of the data's spread each one explains, and you keep only the top few.

Variance, covariance, and the covariance matrix

To make "spread" precise, you need variance. For a single variable x with n observations and mean x̄, the (population) variance is

Var(x) = (1/n) * Σ (x_i - x̄)²

It measures the average squared deviation from the mean — how far, on average, values wander from the center. When you have two variables x and y, you also need to know whether they wander together. That is covariance:

Cov(x, y) = (1/n) * Σ (x_i - x̄)(y_i - ȳ)

If high-x students tend to be high-y students too, the products (x_i − x̄)(y_i − ȳ) are mostly positive (both factors positive together, or both negative together), so the covariance is positive. If large x tends to pair with small y, the covariance is negative. If there's no relationship, the positive and negative products cancel out and covariance is near zero.

For p variables, you don't compute one number — you compute a p × p table called the covariance matrix, where entry (i, j) is Cov(variable i, variable j). The diagonal entries are ordinary variances (Cov(x, x) = Var(x)), and the matrix is symmetric because Cov(x, y) = Cov(y, x). This single matrix is the entire input PCA needs — every principal component is derived purely from this matrix.

Worked example: two subjects, four students

Work through PCA completely by hand on a tiny, honest dataset — four students, two subjects (Maths and Physics marks, out of 100), so every arithmetic step is checkable.

Step 1 — Center the data. PCA always starts by subtracting the mean of each column, so the cloud of points is centered at the origin. This matters because variance and covariance are defined around the mean, and it also means the first principal component will pass through the origin of the centered data.

Mean Maths = (68+66+64+62)/4 = 65. Mean Physics = (68+64+66+62)/4 = 65. Both means happen to be 65. Subtracting gives the centered coordinates:

A: (3, 3)
B: (1, -1)
C: (-1, 1)
D: (-3, -3)

Step 2 — Build the covariance matrix. Using the population formulas above with n = 4:

Var(Maths)  = (3² + 1² + (-1)² + (-3)²)/4 = (9+1+1+9)/4 = 20/4 = 5
Var(Physics)= (3² + (-1)² + 1² + (-3)²)/4 = (9+1+1+9)/4 = 20/4 = 5
Cov(Maths,Physics) = (3·3 + 1·(-1) + (-1)·1 + (-3)·(-3))/4
                   = (9 - 1 - 1 + 9)/4 = 16/4 = 4

So the covariance matrix is

C = | 5  4 |
    | 4  5 |

As a sanity check, the correlation coefficient ρ = Cov(Maths,Physics) / √(Var(Maths)·Var(Physics)) = 4/√(5·5) = 4/5 = 0.8 — a strong positive correlation, matching the visual sense that these two subjects move together.

Step 3 — Find the eigenvalues. This is the step that actually finds the principal axes, so it deserves a full derivation, not a quoted formula. A direction in the plane is a unit vector v. The variance of the data projected onto that direction turns out (this is a short, standard linear-algebra result) to equal vᵀCv. PCA wants the direction that maximizes this projected variance, subject to v being a unit vector (‖v‖ = 1, otherwise you could cheat by making v arbitrarily long). Using a Lagrange multiplier λ to enforce the constraint and differentiating v^T C v - λ(v^Tv - 1) with respect to v and setting the result to zero gives the condition

C v = λ v

This is the defining equation of an eigenvector: a direction that the matrix C stretches without rotating, and λ (the eigenvalue) is exactly the stretch factor — which, tracing back through the derivation, is exactly the variance of the data along that direction. So: the directions of maximum (and minimum) variance are eigenvectors of the covariance matrix, and the amount of variance each one captures is its eigenvalue. This is the single most important fact in PCA, and it is why "compute the eigenvectors of the covariance matrix" is not a mysterious incantation — it falls directly out of maximizing variance under a length constraint.

To find λ, rewrite Cv = λv as (C − λI)v = 0. For a non-zero vector v to solve this, the matrix (C − λI) must be singular, i.e. its determinant must vanish:

det(C - λI) = det | 5-λ    4   | = (5-λ)(5-λ) - 4·4 = 0
                   |  4    5-λ |

(5-λ)² - 16 = 0
25 - 10λ + λ² - 16 = 0
λ² - 10λ + 9 = 0

This is called the characteristic equation. Factor it: (λ − 1)(λ − 9) = 0, so λ = 1 or λ = 9. Two eigenvalues for a 2×2 matrix — exactly as many as the number of original dimensions, which will always be true.

Step 4 — Find the eigenvectors. For each eigenvalue, solve (C − λI)v = 0 to find its direction.

For λ = 9: (C − 9I)v = 0 gives (5−9)x + 4y = 0, i.e. −4x + 4y = 0, i.e. x = y. Any vector along the line y = x works; normalizing to unit length gives v₁ = (1/√2, 1/√2). This is PC1, the first principal component direction.

For λ = 1: (C − 1I)v = 0 gives (5−1)x + 4y = 0, i.e. 4x + 4y = 0, i.e. x = −y. Normalizing gives v₂ = (1/√2, −1/√2). This is PC2, and notice it is automatically perpendicular to v₁ (their dot product is 1/2 − 1/2 = 0) — eigenvectors of a symmetric matrix belonging to different eigenvalues are always orthogonal, which is exactly why PCA produces a set of mutually perpendicular axes rather than an arbitrary set of directions.

Step 5 — Project the data and read off the explained variance. The PC1 score of a student is the dot product of their centered coordinates with v₁; the PC2 score uses v₂.

Student A (3, 3):   PC1 = (3+3)/√2 = 4.243   PC2 = (3-3)/√2 = 0
Student B (1, -1):  PC1 = (1-1)/√2 = 0       PC2 = (1-(-1))/√2 = 1.414
Student C (-1, 1):  PC1 = (-1+1)/√2 = 0      PC2 = (-1-1)/√2 = -1.414
Student D (-3, -3): PC1 = (-6)/√2 = -4.243   PC2 = 0

Two things should jump out. First, A and D — the students whose Maths and Physics marks moved together in the same direction — land exactly on the PC1 axis with PC2 = 0. B and C — whose marks moved in opposite directions relative to the mean — land exactly on the PC2 axis with PC1 = 0. Second, check that the eigenvalues really do equal the variance of the projected scores: Var(PC1 scores) = (4.243² + 0² + 0² + (−4.243)²)/4 = (18+18)/4 = 9, matching λ = 9 exactly. Var(PC2 scores) = (0² + 1.414² + (−1.414)² + 0²)/4 = (2+2)/4 = 1, matching λ = 1 exactly.

Total variance in the original two columns was Var(Maths) + Var(Physics) = 5 + 5 = 10. PCA has redistributed that same total (9 + 1 = 10, conserved exactly — rotation doesn't create or destroy variance) so that PC1 alone carries 9/10 = 90% of it, and PC2 carries the remaining 10%. This 90% figure is the explained variance ratio of PC1. In a real application, if you decided a single "overall ability" score was good enough for your purpose, you would keep only PC1, discard PC2, and have thrown away just 10% of the original spread while cutting the number of columns in half.

Two equivalent ways to see PCA

The derivation above found the direction that maximizes projected variance. There is a second, geometrically different-looking definition that turns out to be mathematically identical: PCA also finds the line (or plane, or hyperplane) that minimizes the sum of squared perpendicular distances from every data point to that line. Look at the worked example again — the PC1 line is y = x. Measure the perpendicular distance from each point to that line, square it, and sum: for A(3,3) and D(−3,−3) the perpendicular distance is exactly 0 (they sit on the line); for B(1,−1) and C(−1,1) the perpendicular distance is |1−(−1)|/√2 = 1.414, squared is 2. Total sum of squared perpendicular distances to the PC1 line = 0 + 2 + 2 + 0 = 4. No other line through the origin gives a smaller total. This "best-fit line by perpendicular distance" view and the "maximum variance direction" view are the same object, proved equivalent by the Pythagorean theorem applied to each point (squared distance from origin = squared distance along the line + squared perpendicular distance from the line, and the first term is fixed by the data, so minimizing perpendicular distance is the same as maximizing the along-the-line spread). The formal statement that keeping the top k principal components gives the best possible rank-k approximation to the data, in the sense of minimizing total squared reconstruction error, is a classical result known as the Eckart–Young theorem.

The general algorithm, for p dimensions

Nothing changes conceptually going from 2 columns to p columns; only the size of the matrices grows.

1. Center each of the p columns (subtract its mean), and decide whether to standardize (covered in the next section).
2. Compute the p × p covariance matrix.
3. Find its eigenvalues λ₁ ≥ λ₂ ≥ … ≥ λ_p and corresponding unit eigenvectors v₁, v₂, …, v_p. (For p > 2 this is not done by factoring a quadratic by hand — software solves the characteristic polynomial numerically — but the definition is identical: Cv = λv.)
4. Order the eigenvectors by eigenvalue, largest first. These are PC1, PC2, … PC_p, mutually perpendicular by construction.
5. Project the centered data onto the top k eigenvectors to get k new columns (the "PCA scores") that replace the original p columns.
6. The explained variance ratio of the first k components is (λ₁ + … + λ_k) / (λ₁ + … + λ_p). Plotting each λ_i against its rank is called a scree plot, and a common rule of thumb is to keep components up to the point where the curve flattens out ("the elbow"), or up to a target cumulative percentage such as 90% or 95%.

Common misconception: "PCA is just linear regression, rotated"

Students who have just learned least-squares regression often assume the PC1 line in the example above is simply the regression line of Physics on Maths. It is not, and the difference is not cosmetic — it comes from what each method minimizes. Ordinary least-squares regression of y on x minimizes the sum of squared vertical distances (actual y minus predicted y) — it treats x as known and error-free, and y as the only noisy quantity. PCA minimizes the sum of squared perpendicular distances — it treats both variables symmetrically, with no notion of one causing or predicting the other.

In the worked example, the regression line has slope Cov(Maths,Physics)/Var(Maths) = 4/5 = 0.8 (this is the standard least-squares slope formula), so the regression line is y = 0.8x — a genuinely different line from the PCA line y = x. It is tempting to guess that the PCA line, precisely because it minimizes total squared perpendicular distance across all four points, must also place every individual point closer to itself than the regression line does. That guess is false, and it is worth checking why. Points A(3,3) and D(−3,−3) sit exactly on the PCA line (perpendicular distance 0) and are 0.469 units from the regression line, so for them PCA wins easily. But points B(1,−1) and C(−1,1) are 1.414 units from the PCA line and only 1.406 units from the regression line — marginally closer to the regression line, not the PCA line. What PCA guarantees is only the aggregate: summed over all four points, the squared perpendicular distance to the PCA line totals 4.0, while the squared perpendicular distance to the regression line totals 4.39 — PCA wins on the total precisely because it wins big on A and D and loses only slightly on B and C. The lesson is to be precise about what an optimum optimizes: PCA's line is optimal in total squared perpendicular distance across the whole dataset, which does not imply it is closer to every single point than every alternative line is.

The standardization pitfall

The worked example used two columns on the same scale (both marks out of 100), so their raw variances (5 and 5) were directly comparable. Real datasets rarely cooperate. Imagine adding a third column, "hours of self-study per week," ranging roughly from 0 to 10, alongside marks out of 100. The variance of a 0–100 column will typically be dozens or hundreds of times larger than the variance of a 0–10 column, purely because of the units chosen, not because that column carries more genuine information. Since PCA maximizes variance, and the covariance matrix is built directly from raw variances, PC1 would end up dominated almost entirely by whichever column happens to have the largest numeric range — the marks columns would drown out the study-hours column regardless of how informative study hours actually are.

The fix is to standardize each column before running PCA: subtract the mean and divide by the standard deviation, so every column has mean 0 and variance 1. Running PCA on standardized data is mathematically the same as running it on the correlation matrix instead of the covariance matrix. As a rule: if your columns are in genuinely different units or wildly different scales, standardize first; if they are already comparable (as in the marks example, or in the Turk–Pentland face-image application in the next section, where every input is a pixel intensity on the same 0–255 scale), using the raw covariance matrix is defensible and sometimes preferred because it preserves the true relative importance of high-variance features.

When PCA has nothing to offer you

PCA earns its keep only when columns are correlated — that correlation is exactly what lets a smaller number of new axes absorb most of the original spread. If your original columns are already close to uncorrelated (their covariance matrix is close to diagonal), the eigenvectors PCA finds will land close to the original axes themselves, and the "compression" gives you almost nothing — you would need nearly as many components as original columns to capture 90% of the variance. PCA is also a purely linear technique: it looks for straight-line (flat) directions of maximum spread, so if the true structure in your data is curved — for instance, points arranged along a spiral or a horseshoe shape — PCA's straight axes will badly misrepresent it, and non-linear alternatives such as Kernel PCA or t-SNE are better suited (both beyond this chapter's scope). Finally, PCA sacrifices interpretability: PC1 in the worked example is "(Maths + Physics)/√2," a specific weighted blend, not a single named quantity you can explain to a parent as easily as "the Maths mark." When you need every output number to map to one clearly nameable original variable, keep the original columns even if there are more of them.

Seeing it in code

First, the hand computation above reproduced with NumPy, using the same four-student dataset, so every printed number matches what was derived by hand:

import numpy as np

X = np.array([
    [68, 68],   # Student A: Maths, Physics
    [66, 64],   # Student B
    [64, 66],   # Student C
    [62, 62],   # Student D
], dtype=float)

mean = X.mean(axis=0)
print(mean)                    # [65. 65.]

Xc = X - mean                  # center the data
cov = np.cov(Xc.T, bias=True)  # bias=True: divide by N (population), not N-1
print(cov)
# [[5. 4.]
#  [4. 5.]]

eigvals, eigvecs = np.linalg.eigh(cov)   # returns eigenvalues in ASCENDING order
print(eigvals)                 # [1. 9.]
print(eigvecs)
# [[-0.70710678  0.70710678]
#  [ 0.70710678  0.70710678]]

scores = Xc @ eigvecs
print(scores)
# [[ 0.          4.24264069]
#  [-1.41421356  0.        ]
#  [ 1.41421356  0.        ]
#  [ 0.         -4.24264069]]

print(scores.var(axis=0))      # [1. 9.]

Notice that eigh orders eigenvalues from smallest to largest, so the second column of eigvecs (eigenvalue 9) is PC1 and the first column (eigenvalue 1) is PC2 — a common source of off-by-one confusion if you forget to check the ordering before deciding which column is "the important one."

Now the same dataset through scikit-learn's ready-made PCA, which sorts components by variance automatically:

from sklearn.decomposition import PCA

p = PCA(n_components=2)
scores = p.fit_transform(X)

print(p.mean_)                      # [65. 65.]
print(p.components_)
# [[0.70710678 0.70710678]
#  [-0.70710678 0.70710678]]
print(p.explained_variance_)        # [12.         1.33333333]
print(p.explained_variance_ratio_)  # [0.9 0.1]

Two numbers deserve a second look. components_[0] is (0.7071, 0.7071) — the same PC1 direction found by hand, up to floating-point rounding. And explained_variance_ is [12, 1.333], not [9, 1] as computed by hand — because scikit-learn divides by N−1 = 3 (the "sample" convention) rather than N = 4 (the "population" convention used in the hand derivation). This rescales both eigenvalues by the same factor of 4/3, so it changes neither the eigenvector directions nor the explained variance ratio, which comes out as [0.9, 0.1] either way — 90% and 10%, exactly matching the hand computation.

Real applications worth knowing

PCA's reach goes far beyond report cards. In 1991, Matthew Turk and Alex Pentland showed that face images, treated as very long vectors of pixel intensities, could be compressed dramatically using PCA — the resulting principal component directions, when reshaped back into images, look like ghostly, generic faces, which is why the technique became known as "Eigenfaces." A small number of eigenface coefficients per photo turned out to be enough to distinguish and recognize individual faces reasonably well, a foundational result in early automated face recognition.

In population genetics, John Novembre and colleagues published a striking result in Nature in 2008: running PCA on genetic variation data from thousands of Europeans, using nothing but genetic markers as input columns (no geographic information at all), produced a PC1–PC2 scatter plot whose shape closely resembled the actual map of Europe — genetic similarity between nearby populations was strong enough that the top two principal components alone recovered a recognizable geography.

In finance, Robert Litterman and José Scheinkman showed in a 1991 paper that the day-to-day movements of an entire government bond yield curve — dozens of interest rates at different maturities, all correlated with each other — could be summarized by just a few principal components, conventionally interpreted as the curve's overall "level," its "slope," and its "curvature." This remains a standard tool in fixed-income risk management.

And in the 2006–2009 Netflix Prize competition, in which teams competed to improve Netflix's movie-rating predictions, low-rank matrix factorization techniques closely related to PCA — decomposing the giant, mostly-empty user-by-movie ratings matrix into a small number of latent "taste" dimensions — were central to the winning approaches, cementing PCA-style dimensionality reduction as a core tool in recommendation systems.

CBSE and competitive-exam mapping

Several pieces of this chapter map directly onto your CBSE Class 11–12 syllabus rather than being extra material bolted on for flavour. Mean, variance, covariance, and the correlation coefficient are exactly the Statistics and Correlation-Regression content in CBSE Mathematics and Applied Mathematics — the ρ = 0.8 computed above is a standard-format numerical answer. The characteristic equation λ² − 10λ + 9 = 0 is an ordinary quadratic, so finding its roots is Class 10 algebra applied in a new context; more generally, setting up and solving det(C − λI) = 0 for a 2×2 (or, with more work, 3×3) matrix draws directly on the Matrices and Determinants chapters of CBSE Class 12, even though the words "eigenvalue" and "eigenvector" are not themselves named in the school syllabus. The Lagrange-multiplier argument used in Step 3 to derive Cv = λv is worth being honest about: it generalizes the single-variable maxima/minima technique you meet in JEE Advanced and BITSAT's Applications of Derivatives to a constrained, multivariable setting — but that generalization itself is not part of the JEE or BITSAT syllabus, and you will meet it formally only in first-year engineering mathematics if you go on to study it there. Treat it here as motivation for why eigenvectors are the right objects to look for, not as exam-ready technique.

Summary

PCA replaces a set of correlated original variables with a smaller set of new, mutually perpendicular variables — the principal components — ranked so that PC1 captures the largest possible share of the data's total variance, PC2 the largest share of what remains, and so on. Mechanically: center the data, compute its covariance matrix, find that matrix's eigenvalues and eigenvectors by solving det(C − λI) = 0 and then (C − λI)v = 0, and project the data onto the eigenvectors with the largest eigenvalues. The eigenvalues themselves report exactly how much variance each component explains, which is what makes "keep the top k components" a quantifiable decision rather than a guess. PCA is not regression — it minimizes total perpendicular distance to a line rather than vertical distance, treating all variables symmetrically — and it is not magic: it needs correlated inputs to have anything to compress, needs standardized inputs when the original columns are on incompatible scales, and only finds straight-line structure, not curved patterns.

Active recall

1. A dataset has covariance matrix [[8, 0], [0, 3]]. Without doing any further calculation, name its two eigenvalues and explain, from the definition of an eigenvector, why the original axes are already the principal component directions here.
2. Explain, in one or two sentences, why the eigenvectors of a covariance matrix belonging to two different eigenvalues are always perpendicular to each other.
3. You run PCA on a table with columns "annual family income in rupees" and "number of siblings," without standardizing first. Predict, with a reason, which column PC1 will end up dominated by.
4. A classmate says, "the PCA line always passes closer to every data point than the regression line does, since PCA is specifically built to minimize distance." Using the worked example's numbers, explain precisely what is wrong with this statement, and what PCA actually guarantees.
5. If the covariance matrix of a 5-column dataset has eigenvalues 40, 30, 20, 8, 2, how many principal components would you need to keep to explain at least 90% of the total variance? Show the running total that gets you there.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where pca: dimensionality reduction wizard is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting pca: dimensionality reduction wizard to at least 3 other topics you have studied.