Linear Algebra for AI: Vectors, Matrices, and Why They Matter

In 2013, researchers at Google trained a language model called Word2Vec on billions of words of text and did something that looked, at first glance, like a magic trick. They took the vector the model had learned for the word "king," subtracted the vector for "man," added the vector for "woman" — and the resulting vector landed almost exactly on the point in space occupied by "queen." No grammar rules were coded in. No dictionary of royal relationships was hand-built. The model had never been told that kings and queens are related the way men and women are. It fell out of pure arithmetic on lists of numbers: subtraction, addition, and distance in a high-dimensional space. That arithmetic is linear algebra, and it is the actual machinery — not a metaphor for the machinery, the literal machinery — underneath almost everything a modern AI system does. A neural network recognizing a face, a recommendation engine deciding what to show you next, a language model predicting the next word: strip away the marketing language and what remains is vectors being added, scaled, and multiplied by matrices, over and over, extremely fast. This chapter builds that machinery from the ground up, precisely enough that you can hand-compute every example yourself.

A vector is a list of numbers with a job to do

Forget "an arrow in space" as your starting definition — that's the second thing a vector is, not the first. The first thing a vector is: an ordered list of numbers, where each position in the list means something specific. Suppose you want to describe a student's performance using three numbers: marks out of 100 in Mathematics, Physics, and Chemistry. If a student scores 92, 88, and 75, you write that as a vector:

v = (92, 88, 75)

This is a vector in 3-dimensional space, written v ∈ ℝ³ (read: "v belongs to real 3-space"). The individual numbers are called components, and the count of components is the dimension. Nothing stops you from having a 300-dimensional vector — a paragraph of text represented by an AI system typically becomes a vector with hundreds of components, one per learned "feature" of meaning. You cannot draw a 300-dimensional arrow, and you don't need to: every rule you're about to learn for 2D and 3D vectors — the ones you can draw — works identically in 300 dimensions. That's the entire reason linear algebra is useful for AI: it gives you a small set of operations that behave the same way no matter how many components a vector has.

The geometric picture is still worth having, because it builds intuition you'll reuse constantly. In 2D, the vector a = (3, 4) is drawn as an arrow starting at the origin (0, 0) and ending at the point (3, 4). Its length, called the magnitude or norm, comes directly from the Pythagorean theorem:

|a| = sqrt(3^2 + 4^2) = sqrt(9 + 16) = sqrt(25) = 5

In general, for a vector with n components, |v| = sqrt(v₁² + v₂² + ... + vₙ²). This single formula scales to any dimension without modification — that's the pattern to get used to.

Adding vectors and scaling them

Vector addition is component-by-component: (a₁, a₂) + (b₁, b₂) = (a₁+b₁, a₂+b₂). If a = (3, 4) and b = (4, 3), then a + b = (7, 7). Geometrically, this is the "tip-to-tail" rule: slide vector b so its tail sits at the tip of a; the sum is the arrow from the origin to b's new tip. In an AI context, addition is how you combine signals — for instance, averaging several users' preference vectors to build a "typical user" vector for a segment.

Scalar multiplication stretches or shrinks a vector without changing its direction (or flips it if the scalar is negative): 2 · (3, 4) = (6, 8), a vector twice as long, same direction. −1 · (3, 4) = (−3, −4), same length, opposite direction. This is how a neural network "turns down" or "turns off" a signal — multiplying a feature vector by a small or negative weight.

The dot product: measuring how aligned two vectors are

Addition and scaling tell you almost nothing about how two vectors relate to each other. For that, AI systems rely overwhelmingly on one operation: the dot product. Given a = (a₁, a₂, ..., aₙ) and b = (b₁, b₂, ..., bₙ), the dot product is defined algebraically as:

a . b = a1*b1 + a2*b2 + ... + an*bn

Multiply matching components, add up the results. For a = (3, 4) and b = (4, 3): a · b = (3)(4) + (4)(3) = 12 + 12 = 24. Notice this produces a single number, not a vector — the dot product always collapses two vectors down to a scalar. That scalar turns out to carry a precise geometric meaning, and it's important enough that we should not just state it — we should derive it, because the whole idea of "similarity" in AI rests on this formula.

Deriving the geometric formula. Draw two vectors a and b from a common origin O, with angle θ between them. They form two sides of a triangle; the third side is the vector connecting the tip of a to the tip of b, which equals b − a, with length |b − a|. The Law of Cosines, applied to this triangle, states:

|b - a|^2 = |a|^2 + |b|^2 - 2|a||b| cos(theta)

Now expand the left side using the dot product's own definition, treating |x|² as x · x:

|b - a|^2 = (b - a).(b - a)
          = b.b - b.a - a.b + a.a
          = |b|^2 - 2(a.b) + |a|^2      (since a.b = b.a)

Set this equal to the Law of Cosines expression, since both describe the same quantity |b − a|²:

|a|^2 + |b|^2 - 2(a.b) = |a|^2 + |b|^2 - 2|a||b| cos(theta)

The |a|² and |b|² terms cancel from both sides, leaving:

-2(a.b) = -2|a||b| cos(theta)
   a.b  = |a| |b| cos(theta)

That's the result, earned rather than asserted: a · b = |a||b|cosθ. Two completely different-looking formulas — one built from components, one built from lengths and an angle — are proven equal. This is why the dot product matters: it's a bridge between "numbers you can compute instantly" (multiply and add components) and "geometry you can reason about" (how aligned two directions are).

Check it against our numbers. a = (3,4), b = (4,3): we computed a·b = 24 algebraically, and |a| = |b| = 5. So cosθ = 24/(5×5) = 24/25 = 0.96, giving θ ≈ 16.3° — a small angle, meaning a and b point in nearly the same direction, which matches the picture below.

Cosine similarity: how AI actually compares two things

The raw dot product mixes together two different pieces of information: how aligned two vectors are, and how long they are. For most AI comparison tasks — is this document about the same topic as that one? does this user's taste resemble that user's taste? — you only want the alignment, not the length. Dividing the dot product by both magnitudes strips the length out, leaving a pure measure of direction called cosine similarity:

cosine_similarity(a, b) = (a . b) / (|a| |b|) = cos(theta)

This always lands between −1 (pointing exactly opposite) and 1 (pointing exactly the same way), with 0 meaning perpendicular — completely unrelated directions. This single formula is the engine behind an enormous fraction of practical AI: search engines ranking documents against your query, recommendation systems finding "more like this," and language models retrieving relevant context all reduce, at some layer, to computing cosine similarity between vectors.

Here's a concrete case. Suppose a video-tutorial platform represents each video as a vector counting how many times it mentions three topic keywords — (algebra, geometry, biology):

A = (4, 1, 0)     "Intro to Algebra" - a 4-minute clip
B = (3, 2, 0)     "Algebra and Geometry Basics" - a 9-minute clip
F = (1, 0.25, 0)  "Quick Algebra Recap" - a 40-second clip

F's topic mix is exactly proportional to A's — F is literally 0.25 × A, so it covers the exact same blend of algebra and geometry, just more briefly. B covers a noticeably different blend (relatively more geometry). If the platform wants to recommend "more videos like A," which should rank higher: B or F?

import numpy as np

def cosine_sim(u, v):
    return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))

A = np.array([4, 1, 0])
B = np.array([3, 2, 0])
F = np.array([1, 0.25, 0])

print(np.dot(A, B), round(cosine_sim(A, B), 2))   # 14   0.94
print(np.dot(A, F), round(cosine_sim(A, F), 2))   # 4.25  1.0

Trace the arithmetic by hand to confirm the code isn't lying to you. Dot products: A·B = 4(3)+1(2)+0(0) = 14. A·F = 4(1)+1(0.25)+0(0) = 4.25. Magnitudes: |A| = √(16+1) ≈ 4.123, |B| = √(9+4) ≈ 3.606, |F| = √(1+0.0625) ≈ 1.031. So cosine_sim(A,B) = 14 / (4.123 × 3.606) ≈ 14 / 14.87 ≈ 0.94, and cosine_sim(A,F) = 4.25 / (4.123 × 1.031) ≈ 4.25 / 4.25 = 1.0 exactly (as it must be, since F is a pure scalar multiple of A, hence perfectly parallel — angle 0°).

Common misconception, corrected: it is tempting to think "bigger dot product means more similar." The numbers above prove that's false. The raw dot product ranks B (14) as more similar to A than F (4.25) — the exact opposite of the truth. B merely has larger components because it's a longer video with more total keyword mentions; that inflates the dot product without meaning the content is more alike. F, despite tiny numbers, is a perfect topical match. This is precisely why real similarity search systems normalize by magnitude — they use cosine similarity, never the raw dot product, whenever the "length" of a vector (document length, number of ratings, total activity) is a side effect of the data rather than a meaningful signal.

Matrices: a grid of numbers that moves every vector at once

A matrix is a rectangular grid of numbers, written with rows and columns. A matrix with m rows and n columns is called an m×n matrix. Matrices matter in AI for a specific, non-decorative reason: a matrix is a machine for transforming vectors, and it transforms every vector in space using the exact same rule. Consider the 2×2 matrix:

M = [ 1  1 ]
    [ 0  1 ]

Matrix-vector multiplication is defined so that each output component is the dot product of a row of M with the vector. For v = (v₁, v₂):

M v = ( 1*v1 + 1*v2 ,  0*v1 + 1*v2 )

There's a more powerful way to see what M does, and it's the single most useful fact in this chapter: the columns of M tell you exactly where the basis vectors e₁=(1,0) and e₂=(0,1) land after the transformation. Compute directly: M·e₁ = (1·1+1·0, 0·1+1·0) = (1, 0) — unchanged. M·e₂ = (1·0+1·1, 0·0+1·1) = (1, 1) — tilted over. Since any vector v = (v₁, v₂) can be written as v₁·e₁ + v₂·e₂, and matrix multiplication respects addition and scaling, you get M·v = v₁(Me₁) + v₂(Me₂) for free — a matrix multiplication is just a weighted combination of its own columns. Check it for v = (2, 1): M·v should equal 2·(1,0) + 1·(1,1) = (2,0)+(1,1) = (3,1). Direct computation confirms it: Mv = (1·2+1·1, 0·2+1·1) = (3, 1). Same answer, two routes — that agreement is not a coincidence, it's the definition working as designed.

This particular M is called a shear: it keeps the x-axis fixed and tilts everything above it sideways, turning a square into a parallelogram, as shown below.

Multiplying two matrices: composing two transformations

If M shears space and R rotates it, what single matrix does "shear, then rotate"? The answer is a new matrix, computed by matrix multiplication. For A (an m×n matrix) and B (an n×p matrix), the product AB is defined entry by entry as:

(AB)[i][j] = row i of A . column j of B

Let A = [[1,1],[0,1]] (our shear) and B = [[1,0],[1,1]] (a shear in the other direction). Compute AB, taking dot products of A's rows with B's columns:

AB row1: (1*1+1*1, 1*0+1*1) = (2, 1)
AB row2: (0*1+1*1, 0*0+1*1) = (1, 1)
AB = [ 2  1 ]
     [ 1  1 ]

Now compute BA the other way round:

BA row1: (1*1+0*0, 1*1+0*1) = (1, 1)
BA row2: (1*1+1*0, 1*1+1*1) = (1, 2)
BA = [ 1  1 ]
     [ 1  2 ]

AB ≠ BA. This is the second common misconception worth naming directly: students who are fluent in ordinary number multiplication (where 3×5 = 5×3) often assume matrix multiplication also doesn't care about order. It very much does. "Apply B, then apply A" and "apply A, then apply B" are, in general, physically different operations that land vectors in different places — exactly the way "put on socks, then shoes" differs from "put on shoes, then socks." In a neural network, this is precisely why the order of layers changes what the network computes: layer order is not a bookkeeping detail, it's mathematically load-bearing.

Where this becomes an actual neural network

A single layer of a neural network computes exactly one line of linear algebra: y = Wx + b, where x is the input vector, W is a weight matrix, b is a bias vector, and y is the output vector — immediately followed by a nonlinear function like ReLU (which simply replaces any negative number with 0). Suppose a tiny network takes two inputs — x = (1, 2), representing, say, ("hours revised today," "practice questions attempted," scaled to convenient units) — and has 3 neurons in its first layer:

W = [ 0.5  -0.2 ]      b = ( 0.1, -0.3, -0.9 )
    [ 0.3   0.8 ]
    [-0.1   0.4 ]

y_pre = W x + b:
  row1: 0.5(1) + -0.2(2) = 0.1   ; +0.1  = 0.2
  row2: 0.3(1) +  0.8(2) = 1.9   ; -0.3  = 1.6
  row3: -0.1(1) + 0.4(2) = 0.7   ; -0.9  = -0.2

y_pre = (0.2, 1.6, -0.2)
ReLU(y_pre) = (0.2, 1.6, 0)      # negative entry clipped to 0

Every arrow of the diagram you drew for the shear matrix M is doing the same job here, just in 2-to-3-dimensional space instead of 2-to-2: each row of W is a dot product being taken against the input, exactly the Σaₖbₖ operation from the very first section of this chapter. Stack several such layers — feed y as the x of the next layer, with its own W and b — and you have a full neural network. There is no additional secret ingredient; depth is just repeated matrix-vector multiplication interleaved with a nonlinearity. This is also, concretely, why GPUs matter for AI: a GPU is hardware built to perform enormous numbers of the multiply-and-add operations that matrix multiplication needs, in parallel — the same (AB)[i][j] = row·column computation from the previous section, done millions of times per second across millions of numbers.

Where this shows up in your exams

Matrices & Determinants and Vectors & 3-D Geometry are formally CBSE Class 12 units, but they are exactly the toolkit tested — often in disguised, multi-step form — across JEE Main and JEE Advanced (angle between vectors, matrix rank and invertibility, transformation geometry), and in BITSAT's mathematics section, which leans heavily on quick vector and matrix manipulation under time pressure. Programs that historically tested this kind of reasoning through pure Olympiad-style problems, such as KVPY before it was folded into other DST fellowship schemes, favoured exactly the "derive it, don't memorise it" approach used above — the law-of-cosines derivation of a·b = |a||b|cosθ is a legitimate exam-style proof, not just classroom colour, and matrix noncommutativity (AB ≠ BA) is a frequently tested "spot the false statement" trap in objective-type questions.

Check your understanding

• Compute a · b for a = (2, −1, 3) and b = (0, 4, 1), then find |a| and |b|, and use them to find cosθ between the vectors.
• Two purchase-history vectors are P = (10, 0, 2) and Q = (5, 0, 1) over categories (books, electronics, stationery). Without computing anything, predict what cosine_similarity(P, Q) will be, and explain why using the "scalar multiple" idea from the F example — then verify by computing it.
• For M = [[2, 0], [0, 3]] (a "scaling" matrix), compute M·e₁ and M·e₂ and describe in one sentence what this matrix does to the unit square, compared to the shear matrix in this chapter.
• Using A = [[1,2],[3,4]] and B = [[0,1],[1,0]], compute both AB and BA by hand and confirm they differ — state, in words, which single entry differs and why.
• A single-neuron layer has weights w = (0.4, −0.6, 0.1) and bias b = 0.2. For input x = (2, 1, 3), compute the pre-activation value w·x + b, then apply ReLU to it.
• Conceptual: explain, in your own words, why cosine similarity rather than the raw dot product is the right tool for comparing a 30-second video's topic vector against a 3-hour lecture's topic vector.

Summary

• A vector is an ordered list of numbers (components); its magnitude is |v| = √(sum of squares of components), generalising Pythagoras to any dimension.
• Vector addition combines signals component-wise; scalar multiplication rescales a vector without changing its direction (except sign flips).
• The dot product a·b = Σaₖbₖ equals |a||b|cosθ — a fact derivable from the Law of Cosines, not just assertable — making it the bridge between raw numbers and geometric alignment.
• Cosine similarity = (a·b)/(|a||b|) isolates direction from magnitude; it, not the raw dot product, is the correct similarity measure whenever vector length reflects an incidental quantity like document length or activity volume.
• A matrix transforms every vector in space by the same rule; its columns are exactly the images of the basis vectors, and matrix-vector multiplication is a weighted sum of a matrix's own columns.
• Matrix multiplication composes transformations and is generally not commutative: AB ≠ BA, because order of operations changes the outcome.
• A neural network layer is literally y = Wx + b followed by a nonlinearity; stacking layers is repeated matrix-vector multiplication, which is why this chapter's arithmetic is not background theory but the actual computation modern AI runs.

Think About It

Think about this: How would you explain linear algebra for ai: vectors, matrices, and why they matter to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.