Singular Value Decomposition (SVD) Simplified

Take any 2×2 matrix and look at what it does to the unit circle — the set of all points exactly 1 unit from the origin. Multiply every point on that circle by the matrix, and the circle never comes out looking like a random blob. It always comes out as an ellipse. Not approximately an ellipse — an exact ellipse, every single time, for every matrix you could write down. That is a strange and specific fact. A matrix can shear, rotate, flip, stretch a shape in complicated-looking ways, and yet the boundary of a perfect circle under that matrix always lands on a perfect ellipse. Singular Value Decomposition is the theorem that explains exactly why this happens, and it hands you three numbers and two directions that describe the ellipse completely: how long its axes are, and which way they point. Once you have those, you have taken the matrix apart into pieces so simple that a rotation, a stretch, and another rotation are literally all that is left.

What the picture is actually claiming

On the left is the unit circle with two special perpendicular directions drawn on it, labelled v₁ and v₂. On the right is the same circle after every point has been multiplied by the matrix A = [[3, 0], [4, 5]] (the exact matrix this chapter works with throughout). The circle became an ellipse, and the directions v₁ and v₂ became two new directions u₁ and u₂ — still perpendicular to each other, even though the matrix itself has no obvious symmetry. The lengths those directions got stretched to are 6.708 and 2.236. Those two stretch factors are called the singular values of A, usually written σ₁ and σ₂ (sigma), always listed largest first. This is the entire geometric content of SVD: every matrix, applied to a circle (or a sphere, in higher dimensions), rotates it, stretches it along perpendicular axes by the singular values, and the result is an ellipse (or ellipsoid). SVD is simply the algebra that finds v₁, v₂, u₁, u₂, σ₁, and σ₂ for any matrix you hand it.

The formal statement

For any real matrix A of size m×n, there exist an m×m orthogonal matrix U, an n×n orthogonal matrix V, and an m×n diagonal matrix Σ with non-negative entries σ₁ ≥ σ₂ ≥ ... ≥ 0 down its diagonal, such that

A = U Σ Vᵀ

"Orthogonal matrix" means its columns are unit vectors that are all mutually perpendicular — an orthogonal matrix Q satisfies QᵀQ = I, which also means Q⁻¹ = Qᵀ (you never need to compute an inverse; the transpose already is the inverse). The columns of V are called the right singular vectors (v₁, v₂, ... — these are the axes on the input circle), the columns of U are the left singular vectors (u₁, u₂, ... — the axes on the output ellipse), and the diagonal entries of Σ are the singular values. Unlike eigenvalues, singular values are defined for every matrix — square, rectangular, symmetric, or not — and they are always real and never negative. That last property is what makes SVD strictly more general-purpose than eigenvalue decomposition, which only exists (in the real numbers, with real eigenvectors) for a narrower class of matrices.

Deriving SVD instead of just stating it

Don't take A = UΣVᵀ on faith — it falls out of one clean argument, and walking through it is what makes the worked example later actually make sense instead of feeling like a recipe.

Start from AᵀA. This matrix is always square (n×n) even when A itself is not, and it has two properties worth naming:

• It is symmetric. (AᵀA)ᵀ = AᵀAᵀᵀ = AᵀA, identical to itself.
• It is positive semi-definite. For any vector x, xᵀ(AᵀA)x = (Ax)ᵀ(Ax) = ‖Ax‖² ≥ 0, a sum of squares, which can never be negative.

This chapter now leans on one major result about symmetric matrices that you have not necessarily proved yet, called the Spectral Theorem: every real symmetric matrix has only real eigenvalues, and its eigenvectors can always be chosen to form a full orthonormal basis. Treat it here as a standard, foundational theorem of linear algebra being introduced for the first time — the way you would accept a new geometry theorem stated cleanly before working through every step of its proof. Its full proof belongs to a later, more advanced course; what matters right now is using it correctly, and it applies directly to AᵀA because AᵀA is symmetric.

So AᵀA has real eigenvalues λ₁ ≥ λ₂ ≥ ... ≥ 0 (non-negative, because positive semi-definiteness rules out negative eigenvalues — if AᵀAv = λv, then λ = λ(vᵀv) = vᵀ(AᵀAv) = ‖Av‖² ≥ 0) with orthonormal eigenvectors v₁, v₂, .... Define the singular values as σᵢ = √λᵢ, and collect the vᵢ as the columns of V. That gives you V and Σ already. Now define, for every σᵢ ≠ 0:

uᵢ = (1 / σᵢ) · A vᵢ

Claim: these uᵢ are orthonormal too. Check the dot product of uᵢ and uⱼ for i ≠ j:

uᵢ · uⱼ = (A vᵢ / σᵢ) · (A vⱼ / σⱼ)
        = (vᵢᵀ Aᵀ A vⱼ) / (σᵢ σⱼ)
        = (vᵢᵀ (λⱼ vⱼ)) / (σᵢ σⱼ)     [since A(A vⱼ) uses AᵀA vⱼ = λⱼ vⱼ]
        = λⱼ (vᵢ · vⱼ) / (σᵢ σⱼ)
        = 0                            [because vᵢ · vⱼ = 0 for i ≠ j]

and for i = j, uᵢ · uᵢ = (vᵢᵀ AᵀA vᵢ)/σᵢ² = λᵢ(vᵢ·vᵢ)/σᵢ² = λᵢ/σᵢ² = 1, since σᵢ² = λᵢ by definition. So the uᵢ really are orthonormal — collect them as the columns of U. Finally, by construction A vᵢ = σᵢ uᵢ for every i, which in matrix form is exactly A V = U Σ. Since V is orthogonal, V⁻¹ = Vᵀ, so multiplying both sides on the right by Vᵀ gives A = U Σ Vᵀ. That is the entire derivation — nothing in it was asserted without justification, and every piece (V, Σ, U) was built explicitly out of AᵀA.

Worked example, start to finish

Let A = [[3, 0], [4, 5]]. Follow the derivation above with real numbers.

Step 1 — Compute AᵀA.

Aᵀ = [[3, 4],
      [0, 5]]

AᵀA = [[3, 4],    [[3, 0],   = [[3·3+4·4, 3·0+4·5],   = [[25, 20],
       [0, 5]]  ·  [4, 5]]      [0·3+5·4, 0·0+5·5]]      [20, 25]]

Step 2 — Find its eigenvalues. AᵀA is symmetric (25/20/20/25, mirrored across the diagonal — exactly as guaranteed), so the Spectral Theorem applies. Its trace is 25 + 25 = 50 and its determinant is 25·25 − 20·20 = 625 − 400 = 225. Both quantities equal, respectively, the sum and the product of the two eigenvalues, giving the characteristic equation:

λ² − 50λ + 225 = 0
λ = (50 ± √(2500 − 900)) / 2 = (50 ± √1600) / 2 = (50 ± 40) / 2
λ₁ = 45,   λ₂ = 5

Both non-negative, exactly as the positive-semi-definiteness argument promised.

Step 3 — Singular values. σ₁ = √45 = 3√5 ≈ 6.7082, σ₂ = √5 ≈ 2.2361. These are precisely the two stretch factors drawn in the diagram above.

Step 4 — Eigenvectors of AᵀA, normalized, become V. For λ₁ = 45: (25−45)x + 20y = 0 → −20x + 20y = 0 → x = y, giving the unit vector v₁ = (1/√2, 1/√2). For λ₂ = 5: (25−5)x + 20y = 0 → 20x + 20y = 0 → x = −y, giving v₂ = (1/√2, −1/√2). Quick check: v₁ · v₂ = 1/2 − 1/2 = 0, confirming they are perpendicular, as the Spectral Theorem guarantees for eigenvectors belonging to different eigenvalues of a symmetric matrix.

Step 5 — Build U from uᵢ = A vᵢ / σᵢ.

A v₁ = [[3,0],[4,5]] · (1/√2, 1/√2) = (3/√2, 9/√2)
u₁   = (3/√2, 9/√2) / (3√5) = (1/√10, 3/√10) ≈ (0.3162, 0.9487)

A v₂ = [[3,0],[4,5]] · (1/√2, −1/√2) = (3/√2, −1/√2)
u₂   = (3/√2, −1/√2) / √5 = (3/√10, −1/√10) ≈ (0.9487, −0.3162)

Check: u₁ · u₂ = 3/10 − 3/10 = 0. Perpendicular, as derived. That completes A = UΣVᵀ by hand:

U ≈ [[0.3162,  0.9487],      Σ = [[6.7082,      0],     Vᵀ ≈ [[0.7071,  0.7071],
     [0.9487, −0.3162]]           [     0, 2.2361]]           [0.7071, −0.7071]]

One subtlety worth naming explicitly: singular vectors are only unique up to a simultaneous sign flip — you can replace (uᵢ, vᵢ) with (−uᵢ, −vᵢ) and A = UΣVᵀ still holds, because the two minus signs cancel in the product σᵢ uᵢ vᵢᵀ. That is exactly why the numbers your computer prints in the next section come out negative even though the by-hand version above used the positive sign convention — both are correct decompositions of the same A.

Verifying with code

NumPy's np.linalg.svd happens to pick the opposite sign convention from the hand computation above, but the magnitudes match exactly. Rounding to 4 decimal places makes the printed output easy to compare against the by-hand values:

import numpy as np

A = np.array([[3, 0],
              [4, 5]])

U, S, Vt = np.linalg.svd(A)

print(np.round(S, 4))
print(np.round(U, 4))
print(np.round(Vt, 4))

# [6.7082 2.2361]
# [[-0.3162 -0.9487]
#  [-0.9487  0.3162]]
# [[-0.7071 -0.7071]
#  [-0.7071  0.7071]]

S holds the singular values, sorted largest-first exactly as the derivation predicted (6.7082 = 3√5, 2.2361 = √5). Vt is Vᵀ, so its rows are v₁ᵀ and v₂ᵀ — both equal to the hand-computed v₁, v₂ with every sign flipped. U's columns are u₁ and u₂, likewise sign-flipped to match. Multiplying U @ np.diag(S) @ Vt reconstructs A exactly (up to floating-point noise around 1e-15), because the sign flips in U and V always occur in matching pairs and cancel out in the product.

Rank-1 approximation and the Eckart–Young theorem

Writing A = UΣVᵀ out in full instead of as a single matrix product reveals something useful: it is really a sum of simple pieces.

A = σ₁ u₁ v₁ᵀ + σ₂ u₂ v₂ᵀ + ... + σᵣ uᵣ vᵣᵀ

Each term σᵢ uᵢ vᵢᵀ is a rank-1 matrix (an outer product), and because σ₁ ≥ σ₂ ≥ ..., the first term carries more of A's "content" than any other single term. This is the entire mathematical basis of using SVD for compression: keep only the first k terms, and you get the best possible rank-k approximation to A, in a precise sense proved by the Eckart–Young theorem — among every matrix of rank k, the truncated SVD sum minimizes the Frobenius-norm error ‖A − Aₖ‖_F, and that minimum error equals exactly √(σ²ₖ₊₁ + σ²ₖ₊₂ + ...), the size of the singular values you threw away. (A grayscale image is literally a matrix of pixel brightness values, which is why this same idea — keep the largest few singular values, discard the rest — is the classic textbook illustration of SVD-based compression; the underlying inequality is identical regardless of what the matrix represents.)

Check this on the worked example by building the rank-1 approximation A₁ = σ₁u₁v₁ᵀ, keeping only the larger singular value:

u1 = U[:, 0]
v1 = Vt[0, :]
A1 = S[0] * np.outer(u1, v1)
print(np.round(A1, 4))
# [[1.5 1.5]
#  [4.5 4.5]]

error = A - A1
print(np.round(np.linalg.norm(error), 4))
# 2.2361

A₁ = [[1.5, 1.5], [4.5, 4.5]] is genuinely rank 1 — its second row is exactly 3× its first row. Eckart–Young predicts the leftover error's Frobenius norm should equal exactly σ₂ = 2.2361, since that is the only singular value discarded, and the code confirms it precisely. By hand: A − A₁ = [[1.5, −1.5], [−0.5, 0.5]], and ‖A − A₁‖_F = √(1.5² + 1.5² + 0.5² + 0.5²) = √(2.25+2.25+0.25+0.25) = √5 = σ₂. No other rank-1 matrix can do better than this — that is the actual theorem, not just a coincidence of this example.

A misconception worth killing now

It is tempting to assume "singular values of A" and "eigenvalues of A" are just two names for the same thing. They are not, and confusing them causes real errors. Eigenvalues are only defined for square matrices in the first place — a 3×2 matrix has no eigenvalues at all, yet it has perfectly good singular values (two of them, since Σ would be 3×2 with two diagonal entries). Even when A is square, its eigenvalues can be negative, complex, or repeated in ways that don't correspond neatly to any stretch factor, whereas singular values — being square roots of eigenvalues of the positive-semi-definite matrix AᵀA — are always real and never negative, by the argument given in the derivation above. The two concepts coincide only in the special case where A is symmetric and positive semi-definite; there, and only there, the singular values of A equal the eigenvalues of A exactly, because AᵀA = A² and √(λ²) = λ when λ ≥ 0. Outside that special case, always go back to AᵀA (or AAᵀ) to get singular values — never substitute eigenvalues of A itself.

Where SVD sits in your exams

The CBSE Class 12 Matrices and Determinants chapter — determinant properties, minors and cofactors, the adjoint, the inverse via A · adj(A) = |A| I, and solving systems with Cramer's rule — is the direct toolkit this chapter leans on to compute AᵀA and its determinant by hand quickly. That fluency is worth building regardless of SVD, since it is tested on its own.

In JEE Main, JEE Advanced, and BITSAT, Matrices and Determinants questions stay within that same boundary: determinant identities, adjoint/inverse relations, rank via row reduction, and Cramer's rule for the consistency of linear systems. Eigenvalues, eigenvectors, and the characteristic equation det(A − λI) = 0 are not part of the JEE Advanced or BITSAT syllabus — you will not be asked to find an eigenvalue in either exam, so do not over-prepare that specific skill for them.

Eigenvalues and SVD become directly examinable once you reach GATE-level engineering mathematics (the common Engineering Mathematics section shared across CS, EE, and ME papers), which regularly tests eigenvalue computation for 2×2 and 3×3 matrices and properties of symmetric matrices exactly like the ones used in the derivation above. If you are aiming at that track eventually, this chapter's spectral-theorem argument is GATE-syllabus content, encountered a few years early.

In Indian mathematical olympiads (RMO/INMO-style problems), formal matrix-eigenvalue computation is rare — these competitions lean heavily on number theory, combinatorics, and synthetic geometry instead. A "matrix" problem that does appear is far more likely to hinge on a determinant identity or a combinatorial argument than on finding an eigenvalue, so it is not worth specifically drilling eigenvalue problems for olympiad prep; that effort pays off far more reliably on the JEE/GATE track.

Test yourself

1. For B = [[0, 3], [4, 0]], compute BᵀB, find its eigenvalues, and give the two singular values of B. (Hint: BᵀB is already diagonal — no characteristic equation needed.)
2. True or false, with a one-line reason: "If A is a 4×2 matrix, A has 4 singular values."
3. For the worked example's A = [[3, 0], [4, 5]], without recomputing U or V, predict ‖A‖_F (the Frobenius norm of A itself) using only σ₁ and σ₂, then verify it against √(3² + 0² + 4² + 5²).
4. Explain in one sentence why uᵢ · uⱼ = 0 for i ≠ j was not assumed but proved in the derivation section — what earlier fact does that proof actually depend on?

Answers. (1) BᵀB = [[0,4],[3,0]]ᵀ... more directly, BᵀB = [[16,0],[0,9]] (already diagonal, since Bᵀ=[[0,4],[3,0]] and BᵀB=[[0,4],[3,0]]·[[0,3],[4,0]]=[[16,0],[0,9]]), eigenvalues 16 and 9, singular values σ₁=4, σ₂=3. (2) False — a matrix has as many singular values as its smaller dimension, so a 4×2 matrix has exactly 2 singular values, not 4; Σ itself is still 4×2 in shape, but only 2 of its entries are on the "diagonal." (3) ‖A‖_F = √(σ₁² + σ₂²) = √(45 + 5) = √50 ≈ 7.0711; directly, √(9+0+16+25) = √50 — they match because ‖A‖_F² = trace(AᵀA) = sum of AᵀA's eigenvalues = sum of σᵢ². (4) It depends on vᵢ · vⱼ = 0 for i ≠ j, i.e., on the Spectral Theorem guaranteeing that eigenvectors of the symmetric matrix AᵀA belonging to different eigenvalues are automatically orthogonal — without that fact the whole orthogonality chain for the uᵢ collapses.

Summary

Every real matrix A factors as A = UΣVᵀ, where U and V are orthogonal and Σ is diagonal with non-negative entries σ₁ ≥ σ₂ ≥ ... called the singular values. Geometrically, A sends the unit circle (or sphere) to an ellipse (or ellipsoid) whose perpendicular semi-axes point along the columns of U and have lengths equal to the singular values. The decomposition is derived, not assumed: AᵀA is always symmetric and positive semi-definite, so the Spectral Theorem hands you real non-negative eigenvalues λᵢ and orthonormal eigenvectors vᵢ; setting σᵢ = √λᵢ and uᵢ = Avᵢ/σᵢ produces an orthonormal U automatically, and A = UΣVᵀ follows. Writing A as a sum σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + ... and truncating it gives the best possible lower-rank approximation to A, with error controlled exactly by the discarded singular values — the Eckart–Young theorem. Singular values are not eigenvalues in general; they only coincide for symmetric positive-semi-definite matrices. On the exam side, the determinant and matrix-inverse machinery of CBSE/JEE/BITSAT is the prerequisite toolkit for everything here, while eigenvalues and SVD itself belong to the GATE-level syllabus a few years ahead.

Think About It

Think about this: How would you explain singular value decomposition (svd) simplified to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where singular value decomposition (svd) simplified is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting singular value decomposition (svd) simplified to at least 3 other topics you have studied.