Eigenvalues and Eigenvectors for Machine Learning

Take a class of three students and look at just two subjects: Physics and Chemistry. Almost always, a student who scores above average in Physics also scores above average in Chemistry — the two marks move together. If you plotted every student's (Physics, Chemistry) pair on a graph, the cloud of points would not spread out evenly in every direction. It would stretch mostly along one diagonal direction ("overall academic strength") and barely spread at all along the direction perpendicular to it ("Physics-versus-Chemistry lean"). Every recommendation engine, every face-recognition system, every dimensionality-reduction step in machine learning is, underneath, doing exactly this: finding the special directions in which data actually varies, and ignoring the directions it doesn't. Those special directions have a name — eigenvectors — and the amount of spread along each one is its eigenvalue. This chapter builds that idea from a bare matrix multiplication up to the algorithm that runs inside real ML systems.

A matrix as a machine that moves vectors

Before defining anything, watch a matrix act on a few vectors. Let

A = [ 2  1 ]
    [ 1  2 ]

and multiply it by three different vectors: u = (1, 0), v₁ = (1, 1), and v₂ = (1, -1).

A·u  = [2·1 + 1·0, 1·1 + 2·0] = (2, 1)
A·v1 = [2·1 + 1·1, 1·1 + 2·1] = (3, 3)
A·v2 = [2·1 + 1·(-1), 1·1 + 2·(-1)] = (1, -1)

Look closely at what happened to each one. The vector u = (1, 0) pointed straight along the x-axis; after multiplication it became (2, 1), which points in a completely different direction — it got rotated as well as stretched. But v₁ = (1, 1) became (3, 3), which is just 3 × (1, 1): same direction, three times the length. And v₂ = (1, -1) became (1, -1) — completely unchanged. For most vectors, multiplying by A both rotates and rescales them. But v₁ and v₂ are special: A only rescales them. They don't rotate at all. The diagram below makes this visible.

In the left panel, u, v₁ and v₂ are three ordinary arrows from the origin. In the right panel, after multiplying every one of them by A: Au has swung off u's original line entirely — the blue arrow no longer points along the x-axis. But the green arrow 3v₁ is still lying exactly on the diagonal line through v₁, only longer. And the purple arrow is lying exactly on top of v₂ — not even stretched. That is the entire idea of an eigenvector: a nonzero vector whose direction a matrix leaves alone, only rescaling it by some factor. That rescaling factor is the eigenvalue.

The formal definition

For a square matrix A, a nonzero vector v is an eigenvector of A if there exists a scalar λ such that

A v = λ v

The scalar λ is the corresponding eigenvalue. In words: multiplying v by the matrix A gives the exact same result as just multiplying v by the number λ. The matrix, which normally shuffles a vector's direction and magnitude in complicated ways, acts on v like a simple scalar. This is a strong condition — it is why eigenvectors are rare special directions, not something every vector satisfies.

Deriving the characteristic equation

Given a matrix A, how do you actually find its eigenvalues and eigenvectors, rather than guessing them as we did above? Start from the definition and rearrange:

A v = λ v
A v - λ v = 0
A v - λ I v = 0        (insert the identity matrix I so both terms are "matrix times v")
(A - λI) v = 0

This says (A - λI) multiplied by v gives the zero vector. Now here is the key step. If the matrix (A - λI) were invertible, you could multiply both sides by its inverse and get v = (A - λI)⁻¹ · 0 = 0. But we specifically require v to be nonzero — that's part of the definition of an eigenvector (v = 0 trivially satisfies A·0 = λ·0 for every λ, so it's excluded by convention, otherwise every number would be an "eigenvalue"). So for a genuine eigenvector to exist, (A - λI) must fail to be invertible. A square matrix fails to be invertible exactly when its determinant is zero. This gives the characteristic equation:

det(A - λI) = 0

For a general 2×2 matrix A = [[a, b], [c, d]], this expands as:

det( [ a-λ   b   ] ) = (a-λ)(d-λ) - bc = 0
    ( [  c    d-λ ] )

λ² - (a+d)λ + (ad - bc) = 0

Notice the two coefficients: (a + d) is the trace of A (sum of diagonal entries), and (ad - bc) is the determinant of A. By Vieta's formulas for a quadratic x² - (sum of roots)x + (product of roots) = 0, this instantly tells you: the sum of the eigenvalues equals the trace, and the product of the eigenvalues equals the determinant. For a 2×2 matrix, this is a genuine shortcut — you almost never need to expand the full determinant by hand once you know trace and determinant. It is a favourite trick in JEE Advanced and BITSAT matrix questions precisely because it turns a quadratic-solving problem into two mental arithmetic checks.

Worked example: solving for λ and v by hand

Return to A = [[2, 1], [1, 2]]. Trace = 2 + 2 = 4. Determinant = 2×2 - 1×1 = 3. So the characteristic equation is:

λ² - 4λ + 3 = 0
(λ - 3)(λ - 1) = 0
λ = 3  or  λ = 1

This confirms what the diagram already showed us numerically: the eigenvalues are exactly 3 and 1 — matching the ×3 stretch we saw for v₁ and the "unchanged" (×1) result for v₂. Now find the eigenvectors properly, by substituting each λ back into (A - λI)v = 0.

For λ = 3:

(A - 3I) = [ 2-3   1  ] = [ -1   1 ]
           [  1   2-3 ]   [  1  -1 ]

(A - 3I) v = 0  gives:  -v1 + v2 = 0   =>   v1 = v2

Any vector with equal components works, e.g. v₁ = (1, 1). Check: A(1,1) = (2+1, 1+2) = (3,3) = 3(1,1). Correct.

For λ = 1:

(A - 1I) = [ 2-1   1  ] = [ 1  1 ]
           [  1   2-1 ]   [ 1  1 ]

(A - I) v = 0  gives:  v1 + v2 = 0   =>   v1 = -v2

So v₂ = (1, -1) works. Check: A(1,-1) = (2-1, 1-2) = (1,-1) = 1·(1,-1). Correct — this matches the "unchanged" arrow from the diagram exactly.

Notice also that v₁ = (1,1) and v₂ = (1,-1) are perpendicular to each other. That is not a coincidence: A is a symmetric matrix (it equals its own transpose, A = Aᵀ), and a theorem called the spectral theorem guarantees that every real symmetric matrix has real eigenvalues and eigenvectors that can always be chosen mutually perpendicular. Its full proof needs more linear algebra than this chapter, but the fact itself is exactly why machine learning leans so heavily on symmetric matrices — you'll see why in the PCA section below.

When eigenvalues repeat

Every eigenvalue problem you've seen so far had two distinct roots. What happens when the characteristic equation has a repeated root? Two very different things can happen, and the difference matters for ML.

Case 1 — the friendly repeated eigenvalue. Take A = [[3, 0], [0, 3]] (3 times the identity matrix). Trace = 6, determinant = 9, so λ² - 6λ + 9 = 0, i.e. (λ-3)² = 0, giving λ = 3 with multiplicity two. Solving (A - 3I)v = 0 gives [[0,0],[0,0]]v = 0 — every vector satisfies this. Here the eigenspace for λ=3 is the entire plane: every nonzero vector is an eigenvector. This matrix simply scales all of space by 3 in every direction, so there's no special surviving direction to find — all directions survive.

Case 2 — the defective repeated eigenvalue. Now take A = [[3, 1], [0, 3]]. Trace = 6, determinant = 9, same characteristic equation, same repeated root λ = 3. But solve (A - 3I)v = 0:

(A - 3I) = [ 0  1 ]
           [ 0  0 ]

(A - 3I)v = 0  gives:  v2 = 0,  v1 free

This time only vectors of the form (v1, 0) — the single line along the x-axis — are eigenvectors, even though λ=3 is "supposed to" have multiplicity two. There is only one independent eigenvector direction for a doubled eigenvalue. Such a matrix is called defective, and it cannot be diagonalized (a concept you'll meet in the next section). This is precisely why machine learning algorithms that need guaranteed clean eigen-decompositions — like PCA — are built on symmetric matrices: the spectral theorem promises that a real symmetric matrix is never defective, no matter how its eigenvalues repeat.

Why any of this matters for computation: diagonalization

Suppose A has two independent eigenvectors v₁, v₂ with eigenvalues λ₁, λ₂. Build a matrix P whose columns are v₁ and v₂, and a diagonal matrix D with λ₁, λ₂ on the diagonal. It's a short exercise to check that A P = P D, and since P is invertible (its columns are independent), this rearranges to:

A = P D P⁻¹

This is called diagonalizing A. It looks like a cosmetic rewrite, but it has a huge computational payoff. Suppose you need A applied 50 times in a row — this happens constantly in ML: repeated steps of an iterative algorithm, repeated layers sharing structure, or long-run behaviour of a Markov chain. Computing A², A³, ..., A⁵⁰ by brute-force matrix multiplication is expensive. But with A = PDP⁻¹:

Aⁿ = (PDP⁻¹)(PDP⁻¹)...(PDP⁻¹) = P Dⁿ P⁻¹

because every inner P⁻¹P cancels to the identity. And Dⁿ is trivial — for a diagonal matrix, raising it to a power just raises each diagonal entry to that power. A problem that looked like 50 expensive matrix multiplications collapses into raising two numbers to the 50th power. This single fact is the computational engine behind analyzing whether a repeated process (a Markov chain settling to steady state, a numerical algorithm converging or blowing up, a linear recurrence in a model) is stable in the long run — you just check whether the eigenvalues have magnitude less than, equal to, or greater than 1.

Principal Component Analysis: eigenvectors doing real ML work

Return to the Physics/Chemistry example from the opening. Suppose three students scored: A: Physics 72, Chemistry 66; B: Physics 69, Chemistry 66; C: Physics 69, Chemistry 63. The mean Physics score is (72+69+69)/3 = 70, and the mean Chemistry score is (66+66+63)/3 = 65. Subtracting the mean from each score gives the deviations: Physics deviations (2, -1, -1), Chemistry deviations (1, 1, -2).

From these deviations, machine learning builds a covariance matrix — a 2×2 table that records how much each subject varies on its own, and how much the two subjects vary together:

Var(Physics)     = (2² + (-1)² + (-1)²) / 3 = 6/3 = 2
Var(Chemistry)   = (1² + 1² + (-2)²) / 3   = 6/3 = 2
Cov(Phy, Chem)   = (2·1 + (-1)·1 + (-1)·(-2)) / 3 = 3/3 = 1

Covariance matrix C = [ 2  1 ]
                       [ 1  2 ]

This is exactly the matrix A we've been working with all chapter. Its eigenvalues, which we already solved, are λ = 3 and λ = 1, with eigenvectors (1,1) and (1,-1). In the language of PCA, these get names: the eigenvector with the largest eigenvalue is the first principal component (PC1), and it is (1, 1) — the "overall academic strength" direction, exactly as guessed at the start of the chapter. The eigenvalue 3 is the variance of the data along that direction. The second eigenvector (1, -1) is PC2, the "relative Physics-vs-Chemistry lean" direction, with variance 1.

A remarkable fact falls straight out of the trace identity proved earlier: the total variance in the data, Var(Physics) + Var(Chemistry) = 2 + 2 = 4, exactly equals λ₁ + λ₂ = 3 + 1 = 4 — the trace is preserved, just redistributed onto the two eigen-directions. This means PC1 alone accounts for 3/4 = 75% of all the variation in the two-subject data. If you had to summarize a student using just one number instead of two marks, projecting onto PC1 keeps 75% of the real information — that is the essence of dimensionality reduction. Real PCA systems do this with hundreds of correlated measurements (pixel brightnesses in a face photo, or genes in an expression dataset) and keep only the handful of eigenvectors with the largest eigenvalues, discarding directions that carry almost no variance.

The three dots are the students' actual (Physics deviation, Chemistry deviation) points. Notice how the cloud clearly stretches further along the green PC1 line than across the purple PC2 line — the diagram is a direct picture of "λ=3 is bigger than λ=1." This is not a coincidence of these three students; it's the geometric meaning of an eigenvalue: it measures how much the data spreads out along its own eigenvector direction.

How machines actually find eigenvectors: power iteration

Solving det(A - λI) = 0 works fine for a 2×2 or 3×3 matrix by hand, but a real covariance matrix in machine learning might be 10,000×10,000 (e.g. one row/column per pixel in a photograph). Expanding a 10,000-degree polynomial is not an option. Instead, ML systems use an iterative method called power iteration: start with any vector, repeatedly multiply by A, and watch where it heads.

import numpy as np

A = np.array([[2, 1],
              [1, 2]])
v = np.array([1, 0])

for i in range(4):
    v = A @ v
    print(v)

Tracing this by hand: start v=(1,0). Step 1: A@(1,0) = (2·1+1·0, 1·1+2·0) = (2,1). Step 2: A@(2,1) = (2·2+1·1, 1·2+2·1) = (5,4). Step 3: A@(5,4) = (2·5+1·4, 1·5+2·4) = (14,13). Step 4: A@(14,13) = (2·14+1·13, 14+2·13) = (41,40). The code prints:

[2 1]
[5 4]
[14 13]
[41 40]

Look at the ratio of the two components at each step: 2/1 = 2, then 5/4 = 1.25, then 14/13 ≈ 1.077, then 41/40 = 1.025. That ratio is marching steadily toward 1 — meaning the vector's direction is converging toward (1, 1), which is exactly PC1, the eigenvector for the larger eigenvalue λ=3. This is the general behaviour of power iteration: repeated multiplication by A amplifies the component of any starting vector along the dominant eigenvector (the one with the largest |λ|) faster than any other component, so the direction converges to it. Google's original PageRank algorithm is a famous real-world use of this exact method: the "importance" ranking of every web page is the dominant eigenvector of a huge matrix built from the web's link structure, and Larry Page and Sergey Brin computed it at Stanford using power iteration rather than any determinant, because the matrix was far too large for characteristic-equation methods. The same idea — find the eigenvector of a large covariance matrix using iteration rather than algebra — is what lets PCA scale to real image datasets, an approach popularized by the 1991 MIT "eigenfaces" work on face recognition, where each face was expressed as a combination of a handful of the covariance matrix's leading eigenvectors.

Two misconceptions worth correcting

Misconception 1: "An eigenvector is one specific vector, like (1,1)." It isn't — it's a direction. Every nonzero scalar multiple of (1,1), such as (2,2), (-1,-1), or (0.5, 0.5), satisfies A v = 3v equally well, because the defining equation only cares about direction and the scaling factor λ, never magnitude or sign. This is why software libraries typically report eigenvectors normalized to unit length, and why you may see a library return (0.707, 0.707) in one run and (-0.707, -0.707) in another for the very same eigenvalue — both are correct, they're just pointing along the same line in opposite senses.

Misconception 2: "Every real square matrix has real eigenvectors you can draw as arrows." This is false, and it's important to know why. Consider the 90° rotation matrix R = [[0, -1], [1, 0]], which rotates every vector in the plane by a quarter turn. Geometrically, no nonzero real vector can stay on its own line after a 90° rotation — every direction gets moved. Algebraically: det(R - λI) = det([[-λ,-1],[1,-λ]]) = λ² - (-1)(1) = λ² + 1 = 0, giving λ = ±i — genuinely complex eigenvalues, with eigenvectors whose entries involve i as well. There is nothing to draw as a real arrow. This matters for ML because it explains a design choice you'll meet constantly: covariance matrices, Gram matrices, and Hessians of well-behaved loss functions are always built to be symmetric specifically so the spectral theorem guarantees real eigenvalues and perpendicular real eigenvectors — the messy complex-eigenvalue case above simply cannot occur for them.

Practice: test yourself

1. Find the eigenvalues of A = [[4, 2], [1, 3]] using the trace/determinant shortcut, then verify by expanding det(A - λI) directly.
2. For the matrix in Q1, find an eigenvector for the larger eigenvalue, and check your answer by computing Av.
3. Find the eigenvalues of B = [[2, 1], [0, 2]]. Is B defective? Find its eigenspace and explain in one sentence why it only has one independent eigenvector direction despite a repeated eigenvalue.
4. A covariance matrix from a different pair of subjects works out to C = [[5, 3], [3, 5]]. Find its eigenvalues and the eigenvector for the larger one, then state what percentage of the total variance the first principal component explains.
5. Without computing anything, explain why a 45° rotation matrix also has no real eigenvectors, but a 180° rotation matrix does.

Answers. (1) trace=7, det=10, λ²-7λ+10=0, (λ-5)(λ-2)=0, so λ=5 and λ=2; expanding directly, (4-λ)(3-λ)-2 = λ²-7λ+10, confirming the shortcut. (2) (A-5I)v=0 gives [[-1,2],[1,-2]]v=0, so v1=2v2, e.g. (2,1); check A(2,1)=(10,5)=5(2,1). (3) trace=4, det=4, λ²-4λ+4=(λ-2)²=0, so λ=2 (repeated). (B-2I)v=0 gives [[0,1],[0,0]]v=0, so v2=0 and only (v1,0) works — one line, so B is defective; the off-diagonal 1 keeps the second coordinate from being free even though λ is doubled. (4) trace=10, det=16, λ²-10λ+16=0, discriminant=36, λ=(10±6)/2 = 8 or 2; for λ=8, (C-8I)v=0 gives [[-3,3],[3,-3]]v=0 so v1=v2, eigenvector (1,1); PC1 explains 8/(8+2)=80% of the total variance. (5) Any rotation strictly between 0° and 180° moves every real vector off its original line, so it cannot fix a real direction and its eigenvalues come out complex; a 0° rotation is the identity (fixes everything, λ=1 doubled) and a 180° rotation sends every vector to its exact opposite (still on the same line, λ=-1 doubled), so those two special angles are the only ones with real eigenvectors.

Summary

An eigenvector of a matrix A is a nonzero direction that A only stretches or shrinks, never rotates off its own line; the scaling factor is its eigenvalue, found by solving det(A - λI) = 0 and then (A - λI)v = 0 for each root. For 2×2 matrices, the sum of eigenvalues equals the trace and their product equals the determinant — a genuine time-saver, not just a curiosity. Repeated eigenvalues can behave two different ways: a full plane of eigenvectors, or — in defective matrices — only a single surviving direction; symmetric matrices are guaranteed by the spectral theorem to always take the well-behaved path, with real eigenvalues and perpendicular eigenvectors, which is precisely why covariance matrices in PCA are built to be symmetric. The eigenvector with the largest eigenvalue of a covariance matrix is the direction along which data varies most, and that eigenvalue is literally the variance along it — the mathematical basis of dimensionality reduction. For matrices too large to factor by hand, power iteration finds the dominant eigenvector by repeated multiplication alone, which is the actual method behind Google's original PageRank and behind PCA at real image and dataset scale.

Think About It

Think about this: How would you explain eigenvalues and eigenvectors for machine learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.