Linear Regression from Scratch: Your First ML Algorithm

The Two Numbers Hiding Inside Your Electricity Bill

Look at any Indian household electricity bill and you'll find a fixed charge that appears no matter what — a meter rental or minimum demand charge — plus a per-unit rate multiplied by the units (kWh) you consumed that month. If the fixed charge is ₹75 and the rate is ₹6 per unit, then for 120 units consumed, the bill is 75 + 6 × 120 = ₹795. This is a straight line: bill = rate × units + fixed charge, or in the language you already know from coordinate geometry, y = mx + c, where m is the slope (the rate) and c is the y-intercept (the fixed charge).

Now suppose you don't know the rate or the fixed charge — you only have six old bills lying around, each showing units consumed and the amount paid. Can you recover m and c just by looking at the data? If the meter were perfect and there were no rounding, six points would lie exactly on one line, and any two of them would hand you the answer. But real bills have rounding, slab-rate quirks, and the odd late-fee — so your six points will almost lie on a line, but not quite. You need a method that looks at every point at once and finds the single best line through the scatter. That method is called linear regression, and it is the oldest, simplest, and most important algorithm in machine learning — every neural network you'll meet later is, at its core, a much bigger version of the same idea you're about to derive by hand.

From a Line You Know to a Line You Have to Find

In coordinate geometry, a straight line y = mx + c is something you're given — two points, or a slope and an intercept, and you plot it. Linear regression flips the problem: you're given a scatter of points (x₁,y₁), (x₂,y₂), ..., (xₙ,yₙ) and asked to find the m and c that make the best line through them. Because no single line can pass through every point of noisy real data, we rename c to b (the convention in machine learning — you'll also see it written as a "bias" term, and the slope written as a "weight" w, so ŷ = wx + b is the exact same equation as y = mx + c wearing ML clothing) and we call the line's output a prediction, written ŷ (read "y-hat") to distinguish it from the actual observed y.

For each data point, the vertical gap between what actually happened and what the line predicts is called the residual or error:

eᵢ = yᵢ - ŷᵢ = yᵢ - (m·xᵢ + b)

A perfect line would make every eᵢ zero. Real data never allows that, so "best line" has to mean "the line that makes the errors smallest overall." The rest of this chapter is about making that vague sentence mathematically precise — and then deriving, without hand-waving, the exact formula for m and b.

Why We Can't Just Add Up the Errors

The tempting first idea is: pick the line that makes Σeᵢ (the sum of all the residuals) as small as possible. This idea is broken, and seeing why is the first real insight of the chapter. Suppose a line runs exactly through the middle of the scatter, with some points 5 units above it and other points 5 units below it. The positive and negative residuals cancel, and Σeᵢ = 0 — but that's true of almost any line through the "middle" of the cloud, including bad ones. A line that is systematically 3 units too high for half the points and 3 units too low for the other half also gives Σeᵢ = 0. Summing signed errors doesn't punish being wrong; it only checks that your wrongness balances out.

The fix is to make every error positive before adding, so cancellation can't happen. There are two natural ways to do this: take the absolute value |eᵢ|, or square it, eᵢ². Machine learning almost always chooses squaring, for two solid reasons, not just tradition:

• Calculus needs a smooth function. |eᵢ| has a sharp corner at eᵢ = 0 where it isn't differentiable, which breaks the calculus we're about to do. eᵢ² is a smooth parabola everywhere, so we can always find its slope and set it to zero.
• Squaring punishes big mistakes harder. An error of 10 contributes 100 to a squared-error sum but only 10 to an absolute-error sum — squaring makes the line work extra hard to avoid being badly wrong on any single point, which is usually what you want from a predictive model.

Averaging the squared errors over all n points gives the single number we'll try to make as small as possible — the cost function (also called Mean Squared Error, MSE):

J(m, b) = (1/n) · Σ (yᵢ - m·xᵢ - b)²    for i = 1 to n

J is a function of two unknowns, m and b — not of x. For any choice of slope and intercept you plug in, J spits out one number: how bad that particular line is on this particular data. Linear regression is now a clean optimisation problem: find the (m, b) pair that makes J as small as it can possibly be.

Deriving the Optimal Line With Calculus

Because J(m, b) is a smooth bowl-shaped surface (more on the "bowl" shortly), its minimum occurs exactly where both partial derivatives are zero — the same logic as finding a minimum of a single-variable function by setting f'(x) = 0, extended to two variables. We differentiate J with respect to b first, treating m as a constant:

∂J/∂b = (1/n)·Σ 2(yᵢ - m·xᵢ - b)(-1) = -(2/n)·Σ (yᵢ - m·xᵢ - b)

Setting this to zero and clearing the -2/n:

Σyᵢ - m·Σxᵢ - n·b = 0
b = ȳ - m·x̄        ...(1)

where x̄ and ȳ are the ordinary means of the x and y values. This single line already tells us something important: the best-fit line always passes through the point (x̄, ȳ) — the "average point" of the data. Now differentiate with respect to m:

∂J/∂m = -(2/n)·Σ xᵢ(yᵢ - m·xᵢ - b)

Setting this to zero and substituting b from equation (1):

Σxᵢyᵢ - m·Σxᵢ² - (ȳ - m·x̄)·Σxᵢ = 0
Σxᵢyᵢ - ȳ·Σxᵢ = m·(Σxᵢ² - x̄·Σxᵢ)

Since Σxᵢ = n·x̄, the left side is Σxᵢyᵢ − n·x̄ȳ and the right bracket is Σxᵢ² − n·x̄². These two quantities have their own names because you'll use them constantly:

Sxy = Σxᵢyᵢ - n·x̄·ȳ      (how x and y move together)
Sxx = Σxᵢ² - n·x̄²        (how much x varies on its own)

m = Sxy / Sxx             ...(2)

Equations (1) and (2) are the complete closed-form solution — no guessing, no iteration, just arithmetic on the data you already have. If you've done the deviation method for finding the mean in the CBSE Class 10 Statistics chapter (Σ of deviations from an assumed mean), you already have the muscle memory for exactly this kind of sum — Sxy and Sxx are simply that same deviation bookkeeping applied to two variables at once. And if you go on to Class 11 Applied Mathematics or Statistics for Economics, you will meet m again under the name regression coefficient of y on x, written b_yx — it is the identical formula, just a different textbook's notation for the same derivation you just completed.

Worked Example: Predicting Marks From Hours Studied

Six students report their weekly study hours (x) and their marks out of 100 on a unit test (y):

Student:   A    B    C    D    E    F
Hours (x): 2    4    5    6    8    9
Marks (y): 35   45   55   60   75   80

Step 1 — means: x̄ = (2+4+5+6+8+9)/6 = 34/6 = 5.667, ȳ = (35+45+55+60+75+80)/6 = 350/6 = 58.333.

Step 2 — the two sums, computed directly (no need to subtract means point-by-point; the shortcut form Σxy − n·x̄·ȳ is faster and less error-prone by hand):

Σxᵢyᵢ = 2(35)+4(45)+5(55)+6(60)+8(75)+9(80)
       = 70+180+275+360+600+720 = 2205

Σxᵢ²  = 4+16+25+36+64+81 = 226

Sxy = 2205 - 6(5.667)(58.333) = 2205 - 1983.33 = 221.67
Sxx = 226 - 6(5.667)² = 226 - 192.67 = 33.33

Step 3 — solve:

m = Sxy / Sxx = 221.67 / 33.33 = 6.65
b = ȳ - m·x̄ = 58.333 - 6.65(5.667) = 58.333 - 37.68 = 20.65

So the fitted line is ŷ = 6.65x + 20.65. Each extra hour of weekly study is associated with about 6.65 extra marks, and a student who studied zero hours is predicted (by extrapolating the line, which is always a little risky) to score around 20.65.

Step 4 — check the fit. Plugging each xᵢ back in gives predictions and residuals:

x   y(actual)  ŷ(predicted)  residual
2   35         33.95         +1.05
4   45         47.25         -2.25
5   55         53.90         +1.10
6   60         60.55         -0.55
8   75         73.85         +1.15
9   80         80.50         -0.50
                sum of residuals ≈ 0.00

The residuals summing to (essentially) zero is not a coincidence — it falls straight out of equation (1), ∂J/∂b = 0, which we derived above. It is a guaranteed mathematical property of the least-squares line, not something we need to check separately.

To judge how good the fit is, compute R² (the coefficient of determination), which compares the leftover error to the total spread in y:

SSres = Σ(residual)² = 1.10+5.06+1.21+0.30+1.32+0.25 = 9.25
SStot = Σ(yᵢ - ȳ)²   = 1483.33

R² = 1 - SSres/SStot = 1 - 9.25/1483.33 = 0.994

R² = 0.994 means the line explains 99.4% of the variation in marks; only 0.6% is left as unexplained scatter. For a single-variable fit, R² equals r², the square of the Pearson correlation coefficient — here r ≈ 0.997, confirming an almost perfectly linear relationship between hours studied and marks in this (deliberately clean) example. Real classroom data will rarely be this tidy; R² values of 0.5–0.8 are common and still useful.

Two Misconceptions Worth Killing Now

Misconception 1: "The residuals summing to zero is how we find the line." It's backwards. Σeᵢ = 0 is not the goal we optimise for — it is a side-effect that falls out automatically once we minimise J(m,b) and set ∂J/∂b = 0. A line can have residuals that sum to exactly zero and still be a terrible fit (imagine one point wildly above the line and one wildly below, roughly cancelling, while the other four points are scattered randomly). The actual objective is always the sum of squared errors; the zero-sum property is a bonus fact you can use to sanity-check your arithmetic, not a substitute for computing m and b properly.

Misconception 2: "A steep slope means a strong relationship." Slope (m) and strength of relationship (r or R²) measure completely different things and are easy to confuse because both come out of the same calculation. m has units — "marks per hour," "rupees per unit" — and can be any real number depending on the scale of your axes; a slope of 6.65 marks/hour is not "stronger" than a slope of 0.5 cm/day just because 6.65 > 0.5, since they're not even measuring the same kind of quantity. Strength of fit is a separate, unitless number between -1 and 1 (r) or 0 and 1 (R²) that tells you how tightly the points cluster around the line, regardless of the line's steepness. A very steep line can have a low R² (points scattered wildly around a steep trend) and a nearly flat line can have R² = 1 (points sitting exactly on a trend that just happens to rise slowly). Always report both numbers, never let one stand in for the other.

Gradient Descent: Finding the Line Without the Formula

The closed-form solution in equations (1) and (2) is exact and, for one input variable, always preferable. But real ML models often have hundreds or millions of parameters (image models, language models), and the closed-form approach for multiple variables requires inverting a matrix — an operation that becomes computationally expensive and numerically unstable as the number of parameters grows. Instead, virtually all of modern machine learning — including the neural networks you'll meet in later chapters — trains models with an iterative search called gradient descent, and linear regression is the cleanest place to learn how it works, because here we can check its answer against the exact formula we already derived.

Picture J(m, b) as a bowl-shaped surface hovering above the (m, b) plane — it is bowl-shaped (mathematicians say convex) precisely because it's built from squared terms, which is yet another reason squaring was the right choice back when we defined the cost function. Gradient descent starts at some guess (often m=0, b=0) and repeatedly takes a small step in the direction that decreases J the fastest — which is the direction opposite to the gradient (∂J/∂m, ∂J/∂b), the two partial derivatives we already computed:

∂J/∂m = -(2/n)·Σ xᵢ(yᵢ - m·xᵢ - b)
∂J/∂b = -(2/n)·Σ (yᵢ - m·xᵢ - b)

repeat until the values stop changing much:
    m := m - α · (∂J/∂m)
    b := b - α · (∂J/∂b)

α (alpha) is the learning rate — how big a step to take on each update. Trace the very first step by hand, starting from m=0, b=0, on our six-student dataset (so every prediction is 0 and every "error" yᵢ − ŷᵢ is just yᵢ itself):

∂J/∂m = -(2/6)·Σ xᵢyᵢ = -(1/3)(2205) = -735
∂J/∂b = -(2/6)·Σ yᵢ   = -(1/3)(350)  = -116.67

with α = 0.01:
m := 0 - 0.01(-735)   = 7.35
b := 0 - 0.01(-116.67) = 1.17

One step already pushed m close to the true value of 6.65 (it slightly overshoots, which is normal — later steps correct it), while b is still far from its true value of 20.65 and needs many more steps to catch up, because the gradient w.r.t. b was much smaller in magnitude than the gradient w.r.t. m for this data. This asymmetry — one parameter racing ahead while the other crawls — is exactly why choosing a good learning rate is a genuine skill: too large an α and the m-update can overshoot so badly on later steps that it oscillates or diverges instead of settling down; too small an α and b would need thousands of extra steps to arrive. Here is the complete algorithm as runnable Python, with no external libraries:

def predict(x, m, b):
    return m * x + b

def cost(x_vals, y_vals, m, b):
    n = len(x_vals)
    total = 0
    for i in range(n):
        error = y_vals[i] - predict(x_vals[i], m, b)
        total += error ** 2
    return total / n

def gradient_descent(x_vals, y_vals, learning_rate=0.01, epochs=5000):
    m, b = 0.0, 0.0
    n = len(x_vals)
    for epoch in range(epochs):
        dm, db = 0.0, 0.0
        for i in range(n):
            error = y_vals[i] - predict(x_vals[i], m, b)
            dm += -2 * x_vals[i] * error
            db += -2 * error
        dm /= n
        db /= n
        m -= learning_rate * dm
        b -= learning_rate * db
    return m, b

x_vals = [2, 4, 5, 6, 8, 9]
y_vals = [35, 45, 55, 60, 75, 80]
m, b = gradient_descent(x_vals, y_vals)
print(f"m = {m:.2f}, b = {b:.2f}")

Because J(m,b) is convex with a single bowl and no other dips, gradient descent with a small enough, stable learning rate is mathematically guaranteed to converge toward the one global minimum — and that minimum is exactly the (m, b) pair given by the closed-form equations (1) and (2). Running this code with learning_rate=0.01 for a few thousand epochs will drive m and b steadily toward 6.65 and 20.65, closing in a little more each pass, the same way the hand-traced first step already started to. This convergence guarantee — no local minima to get stuck in — is special to linear regression; most of the neural networks you'll study later have bumpy, non-convex cost surfaces where gradient descent can only promise to find a good minimum, not necessarily the best one.

Closed Form vs Gradient Descent — Why Bother With Both?

For one input variable, always prefer the closed-form equations (1) and (2) — they are exact, need no learning rate, and finish in one pass over the data. Gradient descent matters because it generalises. With p input variables (predicting marks from hours studied, sleep, and attendance together, say), the closed-form answer becomes a matrix equation w = (XᵀX)⁻¹Xᵀy, where X is a table of all the inputs. Inverting XᵀX costs roughly O(p³) arithmetic operations and can become numerically unreliable when inputs are highly correlated with each other. Gradient descent sidesteps the matrix inversion entirely, updating all p weights with the same simple loop you just wrote, and it scales to the millions of parameters found in modern models precisely because no step ever requires inverting anything. You will meet this multi-variable version, and the matrix form of the equations, in the next chapter on multiple linear regression.

One more genuinely useful connection: the "best-fit line" you're asked to draw with a ruler in CBSE Physics practicals — for Ohm's Law (V vs I), Hooke's Law (F vs extension), or the simple pendulum (T² vs L) — is asking you, informally, to do exactly the least-squares calculation of this chapter. Next time you draw that line by eye, you now know the actual arithmetic that a correctly-drawn line is approximating, and you can compute the real slope instead of guessing it with a ruler.

Test Yourself

1. A plant's height (cm) is recorded weekly: week x = 1,2,3,4 gives height y = 5,8,11,14. Using equations (1) and (2), find m and b by hand. (Answer: m = 3, b = 2 — this dataset is exactly linear, so it's a good way to check your arithmetic method before trusting it on noisy data.)
2. If a regression fit gives SSres = 40 and SStot = 1000, what is R²? What does that number tell you about the fit? (Answer: R² = 1 − 40/1000 = 0.96 — the line explains 96% of the variation in y.)
3. Two students each fit a line to different data. Line A has slope 12 and R² = 0.30. Line B has slope 2 and R² = 0.95. Which line would you trust more for making a prediction, and why? (Use the distinction from Misconception 2 to answer — slope size alone tells you nothing about reliability.)
4. Explain, using the shape of J(m,b), why gradient descent on plain linear regression can never get stuck in the "wrong" minimum the way it sometimes can on a neural network's cost surface.
5. In the hand-traced first gradient descent step above, why did m move much closer to its final value (6.65) after just one update than b did? Which quantity in the gradient formulas is responsible?

Summary

Linear regression fits a line ŷ = mx + b to scattered data by minimising the mean squared error J(m,b) = (1/n)Σ(yᵢ − mxᵢ − b)². Squaring, rather than summing raw or absolute errors, is required because it keeps J smooth enough to differentiate and penalises large mistakes more heavily. Setting both partial derivatives of J to zero yields an exact closed-form solution, m = Sxy/Sxx and b = ȳ − m·x̄, built from the same deviation-sum bookkeeping you already use for computing means. R² (equal to r² for one input variable) measures how much of the variation in y the line explains, and must never be confused with the slope itself, which only measures rate of change. Gradient descent reaches the same answer iteratively, by repeatedly stepping each parameter in the direction that most reduces the cost, scaled by a learning rate — a method that doesn't scale as neatly as the closed form for one variable, but that becomes indispensable the moment your model has more parameters than you can fit into a matrix inversion, all the way up to the neural networks built later in this course.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where linear regression from scratch: your first ml algorithm is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting linear regression from scratch: your first ml algorithm to at least 3 other topics you have studied.