Principal Component Analysis (PCA)

A coaching institute in Kota runs a weekly JEE mock test. Every student walks away with a row of numbers: Physics score, Chemistry score, Maths score, time spent per section, number of questions attempted, number of silly mistakes, accuracy in Mechanics, accuracy in Organic Chemistry, and so on — easily fifteen or twenty columns per student. The analytics team wants to rank students, plot them on a 2-D leaderboard chart, and spot clusters of "similar" students. But nobody can plot a point in twenty dimensions, and nobody needs to — a student who is strong in Physics is usually also strong in Maths, and a student who attempts more questions usually also makes more careless errors. Most of those twenty columns are not twenty independent pieces of information; they are a handful of real underlying patterns of ability, measured redundantly through many correlated numbers. Principal Component Analysis (PCA) is the linear-algebra technique that finds those underlying patterns exactly, with a guarantee: it finds the small set of new directions that preserve the maximum possible amount of the original spread (variance) in the data. This chapter derives that guarantee from scratch — not by asserting it, but by solving the actual optimisation problem.

Variance along a direction, not just along an axis

You already know the variance of a single list of numbers: if a variable X has values x₁, x₂, …, xₙ with mean x̄, then Var(X) = (1/n) Σ (xᵢ − x̄)². It measures how spread out the values are. Now here is the idea that PCA is built on: variance is not a property of an axis you were handed by whoever collected the data. It is a property of any direction you choose to look along.

Suppose each data point is a 2-D vector xᵢ = (Physics score, Maths score), already shifted so the mean is at the origin (we call this "centering": replace each xᵢ by xᵢ − mean). Pick any unit vector w = (w₁, w₂), with w₁² + w₂² = 1. Projecting a point xᵢ onto the direction w gives a single number, xᵢ · w = w₁xᵢ₁ + w₂xᵢ₂ — literally "how far along the w direction does this student sit." Do this for every student, and you get a new 1-D list of numbers. That list has its own variance, Var(w) = (1/n) Σ (xᵢ · w)². Different choices of w give different amounts of spread. Along some directions the projected points are bunched tightly together (little spread, little information); along one particular direction they are stretched out as far as they can possibly go. PCA's first move is to find that one direction — the direction of maximum variance. It is called the first principal component.

The covariance matrix packages all directional variances at once

Expand Var(w) algebraically for 2-D data:

Var(w) = (1/n) Σ (w₁xi1 + w₂xi2)²
       = w₁² · [(1/n)Σxi1²] + 2w₁w₂ · [(1/n)Σxi1xi2] + w₂² · [(1/n)Σxi2²]

The three bracketed quantities do not depend on w at all — they depend only on the data. Give them names: C₁₁ = (1/n)Σxᵢ₁² is the variance of the first feature, C₂₂ = (1/n)Σxᵢ₂² is the variance of the second feature, and C₁₂ = (1/n)Σxᵢ₁xᵢ₂ is their covariance — a measure of how much the two features move together. Arrange them into a matrix:

C = [ C₁₁  C₁₂ ]
    [ C₂₁  C₂₂ ]     where C₁₂ = C₂₁

This is the covariance matrix. It is always symmetric (C₁₂ = C₂₁ by definition of the sum) and it is exactly the object whose entries are variance-of-each-feature on the diagonal and covariance-between-features off the diagonal. In matrix notation, if we stack the centered data as rows of a matrix X (n rows, 2 columns), then C = (1/n) XᵀX. With this matrix, the directional variance formula above collapses to a single clean expression that generalizes to any number of dimensions:

Var(w) = wT C w

This is the whole game now: among all unit vectors w, find the one that maximizes wᵀCw. That maximum value is the variance of the first principal component, and the w that achieves it is the direction of the first principal component.

Worked example: Maths and Science marks of four students

Take four students with marks out of 100:

Student   Maths(x)   Science(y)
   A         70          60
   B         60          70
   C         40          30
   D         30          40

Mean: x̄ = (70+60+40+30)/4 = 50, ȳ = (60+70+30+40)/4 = 50. Center the data by subtracting the mean from every point:

Student   dx = x-50   dy = y-50
   A          20           10
   B          10           20
   C         -10          -20
   D         -20          -10

Now build the covariance matrix by hand, dividing by n = 4:

C₁₁ = (20²+10²+(-10)²+(-20)²)/4 = (400+100+100+400)/4 = 1000/4 = 250
C₂₂ = (10²+20²+(-20)²+(-10)²)/4 = (100+400+400+100)/4 = 1000/4 = 250
C₁₂ = (20·10 + 10·20 + (-10)·(-20) + (-20)·(-10))/4 = (200+200+200+200)/4 = 800/4 = 200

C = [ 250  200 ]
    [ 200  250 ]

Both features have identical spread (250), and they covary strongly and positively (200) — exactly what you'd expect: a student strong in Maths tends to be strong in Science too.

From "maximize variance" to an eigenvalue equation

We now solve the real optimisation problem: maximize f(w) = wᵀCw subject to the constraint g(w) = wᵀw − 1 = 0 (w must stay a unit vector — otherwise you could make wᵀCw arbitrarily large just by stretching w). This is a constrained optimisation problem, and the standard tool is a Lagrange multiplier. Form the Lagrangian:

L(w, λ) = wTCw − λ(wTw − 1)

At a maximum, the gradient of L with respect to w must vanish. Using the standard matrix-calculus identities ∇w(wᵀCw) = 2Cw (valid because C is symmetric) and ∇w(wᵀw) = 2w:

∇wL = 2Cw − 2λw = 0
        ⇒  Cw = λw

This is remarkable: the direction that maximizes variance must be an eigenvector of the covariance matrix C, and the maximum variance achieved along it equals the corresponding eigenvalue λ. This is not a coincidence or an analogy — it falls straight out of calculus applied to the exact quantity we defined variance to be.

Why does the largest eigenvalue win, specifically? Because C is symmetric, the spectral theorem guarantees it has real eigenvalues λ₁ ≥ λ₂ (in 2-D) with orthonormal eigenvectors q₁, q₂ that form a basis. Write any unit vector as w = c₁q₁ + c₂q₂ with c₁² + c₂² = 1 (since q₁, q₂ are orthonormal). Then wᵀCw = c₁²λ₁ + c₂²λ₂, a weighted average of the two eigenvalues with weights that must sum to 1. This weighted average is largest when all the weight sits on the larger eigenvalue — i.e., c₁ = ±1, c₂ = 0, meaning w = ±q₁. So the eigenvector of the largest eigenvalue is exactly the first principal component, and that eigenvalue is exactly the variance it captures. The eigenvector of the second-largest eigenvalue, forced to be orthogonal to the first (eigenvectors of a symmetric matrix belonging to distinct eigenvalues are automatically orthogonal), is the second principal component — the direction of maximum remaining variance once the first direction has been accounted for. In d dimensions the same argument repeats: the principal components are the eigenvectors of C sorted by eigenvalue, largest first, and they are automatically mutually orthogonal.

Solving the eigenvalue problem for our example

Eigenvalues of C solve det(C − λI) = 0:

det [ 250-λ   200    ] = 0
    [  200     250-λ ]

(250-λ)² − 200² = 0
λ² − 500λ + 22500 = 0        (since (250-λ)²-40000 = λ²-500λ+62500-40000)

λ = [500 ± √(500² − 4·22500)] / 2 = [500 ± √160000] / 2 = [500 ± 400] / 2

λ₁ = 450,   λ₂ = 50

Check: λ₁ + λ₂ = 500 = trace(C) = 250+250 ✓, and λ₁λ₂ = 22500 = det(C) = 250·250 − 200·200 = 62500−40000 = 22500 ✓. This double-check (trace = sum of eigenvalues, determinant = product of eigenvalues) is worth memorizing — it catches most arithmetic slips instantly and is itself a frequently tested property of symmetric/square matrices in JEE matrix-algebra problems.

Eigenvector for λ₁ = 450: solve (C − 450I)v = 0:

[ -200   200 ] [v₁]   [0]
[  200  -200 ] [v₂] = [0]   ⇒  -200v₁+200v₂=0  ⇒  v₁=v₂

Normalized: w₁ = (1/√2, 1/√2) ≈ (0.7071, 0.7071).

Eigenvector for λ₂ = 50: solve (C − 50I)v = 0:

[ 200   200 ] [v₁]   [0]
[ 200   200 ] [v₂] = [0]   ⇒  200v₁+200v₂=0  ⇒  v₁=-v₂

Normalized: w₂ = (1/√2, −1/√2) ≈ (0.7071, −0.7071). Check orthogonality: w₁ · w₂ = (1/√2)(1/√2) + (1/√2)(−1/√2) = ½ − ½ = 0 ✓, exactly as the theory promised.

What the two components mean, and their variance share

PC1's direction (1,1)/√2 weights Maths and Science equally and adds them — it is literally an "overall academic strength" axis. PC2's direction (1,−1)/√2 subtracts them — it is a "Maths-minus-Science skew" axis. Project every centered student onto each:

Student   PC1 = (dx+dy)/√2    PC2 = (dx−dy)/√2
   A       (20+10)/1.414 = 21.21    (20-10)/1.414 =  7.07
   B       (10+20)/1.414 = 21.21    (10-20)/1.414 = -7.07
   C      (-10-20)/1.414 = -21.21   (-10+20)/1.414 =  7.07
   D      (-20-10)/1.414 = -21.21   (-20+10)/1.414 = -7.07

The fraction of total variance captured by each component is its eigenvalue divided by the sum of all eigenvalues: PC1 explains 450/500 = 90%, PC2 explains 50/500 = 10%. If the analytics team keeps only PC1 and throws away PC2, they lose just 10% of the total spread in the data while cutting the dimension in half — from 2 numbers per student down to 1. Notice too that PC1 separates {A,B} (score +21.21) cleanly from {C,D} (score −21.21) — the "strong pair" versus the "weaker pair" — which is exactly the ranking signal a leaderboard needs, extracted automatically from raw marks with no manual weighting formula.

The diagram

The blue PC1 axis runs through the long diagonal spread of the data (A and B sit far up-right, C and D far down-left) — that is the direction of maximum variance the eigenvalue calculation found. The orange PC2 axis is perpendicular to it and only picks up the small leftover spread — how far above or below the diagonal each point sits. This is the geometric picture behind every number computed above.

Verifying with code

The same computation, done by a library instead of by hand:

import numpy as np

X = np.array([[70,60],[60,70],[40,30],[30,40]], dtype=float)
mean = X.mean(axis=0)              # [50. 50.]
Xc = X - mean                      # centered data
cov = np.cov(Xc.T, bias=True)      # bias=True divides by n, matching our hand formula
print(cov)
# [[250. 200.]
#  [200. 250.]]

eigvals, eigvecs = np.linalg.eigh(cov)   # eigh returns eigenvalues ASCENDING
print(eigvals)                     # [ 50. 450.]
print(eigvals / eigvals.sum())     # [0.1 0.9]  -> PC(50) explains 10%, PC(450) explains 90%

Running this prints eigenvalues 50 and 450, in that ascending order, because numpy.linalg.eigh (the routine built specifically for symmetric matrices) is documented to sort ascending — the largest eigenvalue, and hence the true first principal component, is the last column of the returned eigenvector matrix, not the first. This is a real, common bug source: code that blindly takes eigvecs[:, 0] as "the first principal component" is silently taking the direction of least variance instead. (You may also get the eigenvector for eigenvalue 450 back as (0.7071, 0.7071) or as (−0.7071, −0.7071) — both point along the identical line and are equally valid; eigenvectors are only defined up to an overall sign flip, since if Cw = λw then C(−w) = λ(−w) too.)

Why total variance splits exactly: the reconstruction-error view

There is a second, equivalent way to justify PCA that many textbooks state without proof — that principal components also minimize squared reconstruction error when you approximate each point by its projection. Here is why the two views are the same problem in disguise. Because w₁, w₂ are orthonormal, they form a basis, so any centered point xᵢ can be written exactly as xᵢ = (xᵢ·w₁)w₁ + (xᵢ·w₂)w₂. By the Pythagorean theorem (since w₁ ⊥ w₂), ‖xᵢ‖² = (xᵢ·w₁)² + (xᵢ·w₂)². Summing over all points and dividing by n: total variance = Var(w₁) + Var(w₂) = λ₁ + λ₂, a fixed number that does not depend on which orthonormal basis you chose. Now suppose you keep only the PC1 coordinate and reconstruct each point as x̂ᵢ = (xᵢ·w₁)w₁, discarding the PC2 part. The squared reconstruction error is ‖xᵢ − x̂ᵢ‖² = (xᵢ·w₂)², whose average over all points is exactly Var(w₂) = λ₂. So: total variance is fixed, captured variance (λ₁) plus reconstruction error (λ₂) always add up to that fixed total. Maximizing the variance you keep and minimizing the error you introduce are therefore the exact same optimisation — you cannot improve one without improving the other. This is why PCA is simultaneously described as "the direction of maximum spread" and "the best possible linear summary" in every source you will read; both descriptions are proved true by this one identity.

Why you must standardize features on different scales

Our worked example used two features on an identical scale (both marks out of 100), so the covariance matrix directly reflected genuine correlation. Real datasets rarely cooperate. Suppose you added a third feature, "family monthly income in rupees," ranging into tens of thousands, alongside marks that range 0–100. Income's variance, measured in rupees², will be astronomically larger than any mark's variance measured in points², purely because of the unit, not because income is "more important." The covariance matrix would then be dominated by that one large-scale feature, and PC1 would point almost entirely along the income axis regardless of what the marks are doing — an artifact of measurement units, not a genuine pattern in the data. The fix is to standardize every feature first: replace each value by (value − mean)/standard deviation, so every feature has variance exactly 1 before PCA runs. Diagonalizing the covariance matrix of standardized data is the same as diagonalizing the correlation matrix of the raw data — a matrix whose diagonal entries are always 1 and whose off-diagonal entries are correlation coefficients between −1 and 1. Whenever your features are not already in comparable units, standardize first; skipping this step is one of the most common real-world PCA mistakes.

Two misconceptions, corrected

Misconception 1: "PCA picks the most important original columns and drops the rest — like keeping Maths and dropping Science." This is false. Not a single original column survives PCA unchanged. Every principal component is a new variable built as a linear combination of all the original features — in our example, PC1 is literally (Maths + Science)/√2, not "Maths alone." This is precisely why PCA is called a feature extraction technique, in contrast to feature selection techniques (like simply dropping low-variance columns), which do keep a subset of the original, individually meaningful columns. After PCA, you generally cannot point to "the Physics axis" anymore — only to combinations, and interpreting what a component "means" (as we could in our clean 2-feature example) becomes much harder once there are twenty original features mixed together.

Misconception 2: "The direction of highest variance must be the most useful direction for whatever I'm trying to predict." This is also false, and it is PCA's central limitation. PCA is completely unsupervised — its objective function, wᵀCw, never looks at any target label; it only ever looks at the spread of the inputs. It is entirely possible to construct a dataset where the direction that best separates two classes (say, "will clear JEE cutoff" vs. "will not") is the low-variance PC2, while the high-variance PC1 is irrelevant noise that varies a lot but carries no signal about the outcome you actually care about. If your goal is prediction rather than compression or visualization, a supervised alternative such as Linear Discriminant Analysis (which does use the labels, maximizing between-class separation rather than raw variance) can outperform PCA precisely because it optimizes a different, label-aware objective. Always ask what you are actually trying to preserve before assuming "more spread" means "more useful."

A degenerate case worth knowing: zero eigenvalues

Consider a covariance matrix C = [[8,4],[4,2]]. Its trace is 10 and its determinant is 8·2 − 4·4 = 16 − 16 = 0. Since the product of the eigenvalues equals the determinant, one eigenvalue must be exactly 0 — solving λ² − 10λ = 0 gives λ = 0 or λ = 10 directly. A zero eigenvalue means the data has no spread whatsoever along that eigenvector's direction — every point lies exactly on a single line through the mean. The two original features here were never truly two independent pieces of information; they were one genuine underlying quantity measured through two perfectly correlated numbers, and PCA reveals that the data's real dimensionality is 1, not 2. This is the extreme, clean version of the redundancy that PCA is built to find and remove.

Real applications, correctly attributed

Eigenfaces, one of the earliest successful face-recognition systems, was built by Matthew Turk and Alex Pentland at MIT in 1991: they treated each face photograph as one very-high-dimensional vector (one number per pixel) and used PCA to find a small number of "eigenface" directions capturing most of the variance across many faces, representing each new face by just its coordinates along those directions instead of the raw pixels. Gene-expression studies use PCA to compress readings across thousands of genes per sample down to a handful of components that separate healthy from diseased tissue. Remote-sensing platforms that record many spectral bands per pixel use PCA to compress correlated band readings into a few components before further analysis. In every case the underlying justification is the identical mathematics you just derived: build the covariance matrix, diagonalize it, keep the eigenvectors with the largest eigenvalues.

Where this fits your exams

PCA itself is not named in the CBSE Class 10–12 syllabus, but every tool this chapter used is core CBSE and JEE mathematics: determinants and the characteristic equation det(A − λI) = 0 (Class 12 matrices/determinants), solving quadratics for eigenvalues, properties of symmetric matrices (real eigenvalues, orthogonal eigenvectors — a recurring JEE Advanced and KVPY topic on its own, independent of any data-science framing), dot products and unit vectors, and Lagrange-style constrained optimisation (the same reasoning that appears in JEE optimisation-with-constraint problems, just applied to a matrix quadratic form instead of a scalar function). If you sit GATE's Data Science and Artificial Intelligence paper, PCA is directly and explicitly examinable, usually as exactly this kind of small-matrix, compute-the-eigenvalues-by-hand problem — meaning the worked example above is not just an illustration, it is close to the literal exam format.

Active recall

1. A covariance matrix is C = [[3,1],[1,3]]. Find both eigenvalues by solving det(C−λI)=0, find the corresponding unit eigenvectors, and state what percentage of the total variance the first principal component explains.
2. Explain, in one or two sentences and without just citing "the spectral theorem" as a magic word, why the two principal components of any covariance matrix in 2-D must come out mutually orthogonal.
3. A dataset has features "distance run in kilometres" and "time taken in seconds," and you skip standardization before running PCA. Predict, with a reason, which feature will dominate the first principal component, and explain the concrete fix.
4. True or False, with a one-line justification for each: (a) PCA can be computed without ever centering the data first. (b) The eigenvector belonging to the smallest eigenvalue of a covariance matrix is the direction PCA discards first. (c) If two features have zero covariance but different individual variances, the principal components are still guaranteed to be the original coordinate axes.
5. A covariance matrix has trace 20 and determinant 36, and you're told one eigenvalue is 2. Find the other eigenvalue two independent ways — first using the trace, then using the determinant — and confirm they agree.

Summary

• For centered data, the variance captured by projecting onto a unit direction w is wᵀCw, where C = (1/n)ΣxᵢxᵢT is the covariance matrix.
• Maximizing wᵀCw subject to ‖w‖=1 via a Lagrange multiplier gives Cw = λw: the maximizing directions are eigenvectors of C, and the variance each captures equals its eigenvalue.
• Because C is symmetric, its eigenvalues are real and eigenvectors for distinct eigenvalues are automatically orthogonal (spectral theorem) — this is why principal components never overlap in direction.
• Sorting eigenvalues largest-to-smallest and keeping their eigenvectors gives the principal components in order of importance; the fraction of total variance each explains is λᵢ / Σλⱼ.
• Capturing variance and minimizing squared reconstruction error are mathematically the same objective, via the Pythagorean identity total variance = captured variance + discarded variance.
• PCA creates brand-new combined features and never simply selects old ones; it is unsupervised and can discard directions that matter for a specific prediction task even while keeping the highest-variance ones; and it requires standardized inputs whenever original features are on different scales.

Think About It

Think about this: How would you explain principal component analysis (pca) to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where principal component analysis (pca) is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting principal component analysis (pca) to at least 3 other topics you have studied.