Linear Regression by Hand: Understanding the Math

Meera takes an autorickshaw to school every day, and she has started noticing something: the fare depends on the distance, but not in a way she can guess exactly. One day she decides to keep the receipts for a week. Here is what she collects:

If the fare went up by a fixed amount for every kilometre, all five points would sit perfectly on a straight line, and Meera could just read off the "rate per km" and the "base fare" from any two rows. But real data is never that obedient. From 1 km to 2 km the fare jumps by ₹10. From 2 km to 3 km it jumps by only ₹5. From 4 km to 5 km it jumps by ₹10 again. There is clearly a rising trend, but no single straight line passes through all five points exactly. This is the situation linear regression is built for: finding the one straight line that best represents a trend in data that is roughly linear but not perfectly linear. This is also the simplest possible example of what "machine learning" means in practice — using past data (distance and fare pairs Meera already has) to build a rule that predicts a new case (the fare for a distance she hasn't recorded yet).

Why You Can't Just Connect Two Points

A natural first instinct is to pick the first and last data points and draw a line through them, since a line is fully determined by two points. Using (1, 25) and (5, 60), the slope would be (60 − 25) ÷ (5 − 1) = 35 ÷ 4 = 8.75 rupees per km, and plugging back into the first point gives an intercept of 25 − 8.75(1) = 16.25. So this "naive" line is fare = 8.75 × distance + 16.25.

The problem is that this line was built using only 2 of the 5 receipts. The points at 2 km, 3 km, and 4 km had no say in where the line goes at all — they could be wildly off and the naive line would not change one bit. A trustworthy method has to let every data point pull the line toward itself. That is exactly what "least squares" regression, the method you're about to learn, does. Keep this naive line in mind — we will come back and prove, with actual numbers, that it is worse than the line least squares produces.

Measuring How Wrong a Line Is: Residuals

Suppose we have some candidate line, written in the familiar slope-intercept form:

y_predicted = m * x + c

Here m is the slope (how many rupees the fare rises per extra km) and c is the intercept (the fare when distance is 0 — essentially the base/pickup charge). For any data point, we can compare what the line predicts against what actually happened. That difference has a name:

residual = actual_y - predicted_y

A residual of 0 means the line predicted that point perfectly. A large positive residual means the actual value was well above the line; a large negative residual means it was well below. To judge whether one line is "better" than another, we need to combine all five residuals (one per data point) into a single number. Just adding them up is a bad idea, because a residual of +5 at one point and −5 at another would cancel out to 0, making a genuinely bad line look perfect. We need every residual to count against the line regardless of its sign.

Why We Square the Errors

The standard fix is to square every residual before adding them up. Squaring does two useful things at once. First, it removes the sign problem: (−5)² = 25 and (+5)² = 25, so a miss of 5 in either direction is punished equally — no cancellation. Second, it punishes big misses disproportionately more than small ones: a residual of 2 contributes 4, but a residual of 4 (twice as large) contributes 16 (four times as much). This matches how we actually feel about prediction errors — being off by 4 km-fare-rupees feels more than twice as bad as being off by 2. The total is called the sum of squared errors, or SSE:

SSE = (residual_1)^2 + (residual_2)^2 + ... + (residual_n)^2

"Linear regression" is precisely the search for the values of m and c that make SSE as small as possible, over every straight line you could possibly draw. This is why the method is also called least squares — we are looking for the line with the least sum of squares.

The Formula for the Best-Fitting Line

Finding the exact m and c that minimise SSE normally uses calculus, which is beyond what you need right now. What matters at your level is that mathematicians have already done that work and handed us a clean, reusable recipe. If your data points are (x₁, y₁), (x₂, y₂), …, (xₙ, yₙ), and x̄ and ȳ are the mean (average) of the x-values and y-values, then the best-fitting line's slope and intercept are:

m = sum( (x_i - x_mean) * (y_i - y_mean) )  /  sum( (x_i - x_mean)^2 )

c = y_mean - m * x_mean

Don't just memorise this — read what each piece is actually saying. The numerator of m checks, for every point, whether x and y moved together: if a point's x is above average and its y is also above average, that product is positive (positive × positive); if x is above average but y is below average, the product is negative. Adding all these products up tells you the overall direction and strength of the relationship between x and y. The denominator, sum((x_i - x_mean)^2), only involves x, and measures how spread out the x-values are. Dividing "how x and y move together" by "how spread out x is" converts that co-movement into a rate — rupees per kilometre, in Meera's case. Once you know the slope, the intercept formula is just saying: the line must pass through the point (x̄, ȳ), so use that point and the slope you already found to solve for where the line crosses x = 0.

Working the Numbers by Hand

Let's apply this recipe to Meera's five receipts. First, the means:

x_mean = (1 + 2 + 3 + 4 + 5) / 5 = 15 / 5 = 3
y_mean = (25 + 35 + 40 + 50 + 60) / 5 = 210 / 5 = 42

Now build a table of deviations from the mean, their product, and the squared x-deviation, one row per receipt:

Adding the second-to-last column: 34 + 7 + 0 + 8 + 36 = 85. Adding the last column: 4 + 1 + 0 + 1 + 4 = 10. So:

m = 85 / 10 = 8.5
c = 42 - 8.5 * 3 = 42 - 25.5 = 16.5

The best-fitting line is fare = 8.5 × distance + 16.5. In plain words: Meera's autorickshaw driver seems to charge roughly ₹16.50 as a base amount, plus about ₹8.50 for every kilometre travelled. Compare this to the naive "connect the endpoints" line from earlier (8.75x + 16.25) — close, but not the same, because this line was pulled into place by all five receipts, not just two.

Seeing It: Data Points and the Fitted Line

Notice that the point at 1 km sits exactly on the red line — its residual is 0. The points at 3 km and 4 km sit slightly below the line (negative residuals), while the points at 2 km and 5 km sit slightly above it (positive residuals). No line could touch all five points, but this particular line has been placed so that these misses, when squared and added, are as small as they can possibly be for any straight line.

A Guaranteed Property: The Line Passes Through the Average Point

Here's a fact you can check yourself, and it isn't a coincidence — it falls directly out of how the c formula was built. The point (x̄, ȳ) = (3, 42) always lies exactly on the regression line. Check it: 8.5 × 3 + 16.5 = 25.5 + 16.5 = 42. Exactly ȳ. This makes sense once you remember how we derived c: we forced the line to pass through the average point and then solved for c. So no matter what data you feed this recipe, the fitted line is guaranteed to pass through the average of the x-values and the average of the y-values. This is a fast way to sanity-check any regression line you compute by hand — if it doesn't pass through (x̄, ȳ), you've made an arithmetic slip somewhere.

Using the Line to Predict — and the Danger of Extrapolation

The whole point of finding this line is to predict fares for distances Meera hasn't recorded yet. For a 7 km trip:

fare = 8.5 * 7 + 16.5 = 59.5 + 16.5 = 76.0

Predicting 7 km is reasonable because it's close to the range Meera actually measured (1 km to 5 km). But watch what happens if we get greedy and predict a 40 km trip: 8.5 × 40 + 16.5 = 340 + 16.5 = ₹356.50. Maybe that's fine, or maybe autorickshaws refuse trips beyond a certain distance, or switch to a completely different pricing slab for highway stretches. Meera's data says nothing at all about distances beyond what she actually measured. Using a fitted line to predict far outside the range of your original data is called extrapolation, and it is one of the most common ways people misuse regression. The line is a summary of the pattern within the data you collected — trusting it blindly outside that range is a guess dressed up as a calculation.

The Residuals Always Balance Out to Zero

Let's compute every residual for Meera's line and add them up:

Sum of residuals: 0.0 + 1.5 − 2.0 − 0.5 + 1.0 = 0.0. This is not a coincidence specific to Meera's numbers — for the least-squares line, the positive and negative residuals will always cancel out to exactly zero. It's another consequence of the line being forced through the average point: the total amount the line "overshoots" some points by must exactly balance the total amount it "undershoots" others by, otherwise you could nudge c up or down slightly and reduce SSE further, contradicting the fact that we already found the minimum.

Automating It: Linear Regression in Python

Doing the deviation table by hand is good for building intuition, but for real datasets with hundreds of rows you'd write code. Here is a direct translation of the exact formula, using only lists — no external library:

distance = [1, 2, 3, 4, 5]        # kilometres
fare     = [25, 35, 40, 50, 60]   # rupees

n = len(distance)
x_mean = sum(distance) / n
y_mean = sum(fare) / n

numerator = 0
denominator = 0
for i in range(n):
    numerator += (distance[i] - x_mean) * (fare[i] - y_mean)
    denominator += (distance[i] - x_mean) ** 2

m = numerator / denominator
c = y_mean - m * x_mean

print("slope m =", m)
print("intercept c =", c)

Trace it exactly like the harness will: x_mean becomes 15/5 = 3.0 and y_mean becomes 210/5 = 42.0. The loop runs five times, accumulating the same products you already computed by hand — 34, 7, 0, 8, 36 into numerator, reaching 85.0, and 4, 1, 0, 1, 4 into denominator, reaching 10.0. Then m = 85.0 / 10.0 gives 8.5, and c = 42.0 - 8.5 * 3.0 gives 42.0 − 25.5 = 16.5. Because every one of these numbers — 3.0, 42.0, the deviations, 8.5, 16.5 — happens to be a value with a short exact binary fraction (like 8.5 = 17/2 or 16.5 = 33/2), this particular program prints exactly:

slope m = 8.5
intercept c = 16.5

with no odd extra digits, because the running print() uses Python's shortest round-trip representation of the float, and both 8.5 and 16.5 are exactly representable in binary floating point. You can extend the same program with a small prediction function:

def predict_fare(distance_km, m, c):
    return m * distance_km + c

print(predict_fare(7, m, c))   # 8.5 * 7 + 16.5 = 76.0

A Word of Caution About Floating-Point Numbers

It's tempting to conclude from the example above that Python arithmetic is always this clean. It usually is not, and this is worth knowing before you write regression code on your own datasets. Try this in a Python shell:

>>> 0.1 + 0.2
0.30000000000000004

This is a completely different, unrelated fact from anything about our fare example — it happens because 0.1 and 0.2 do not have exact binary representations (much like 1/3 has no exact finite decimal representation), so tiny rounding errors creep in during storage and addition. It matters for you as a regression programmer because if your dataset's numbers do not divide as cleanly as Meera's did, your computed residuals might come out as, say, 4.999999999999998 instead of a clean 5.0. If you ever write code that checks if sum_of_residuals == 0:, that check can silently fail even when the math is correct, purely because of this representation issue. The safe practice is to use a tolerance-based comparison, such as math.isclose(sum_of_residuals, 0, abs_tol=1e-9), rather than testing floats for exact equality.

Common Misconception, Corrected With Numbers

A very common belief is: "the more data points a line touches, the better it is," and a closely related one is: "you can build a good trend line by just connecting two data points that look representative." We can now settle this with an actual number instead of just asserting it. Recall the naive line from the start of this chapter, built by connecting (1, 25) and (5, 60): fare = 8.75 × distance + 16.25. Let's compute its SSE the same way we would for any candidate line:

SSE for the naive line: 0 + 1.5625 + 6.25 + 1.5625 + 0 = 9.375. Now compare that with the SSE of our least-squares line, using the residuals we already found (0.0, 1.5, −2.0, −0.5, 1.0): 0² + 1.5² + (−2.0)² + (−0.5)² + 1.0² = 0 + 2.25 + 4.0 + 0.25 + 1.0 = 7.5. The least-squares line's SSE (7.5) is smaller than the naive line's SSE (9.375), confirming numerically that using all five receipts genuinely produces a better line than cherry-picking two of them — even though the naive line happens to touch two of the actual data points exactly, and the least-squares line touches only one. "Touching more points" is not the goal; "minimising total squared error across every point" is.

Try It Yourself

An ice-cream stall owner records the outside temperature and the number of cones she sells that day:

1. Compute x̄ and ȳ for this data.
2. Build the deviation table (x−x̄, y−ȳ, their product, and (x−x̄)²) and find m and c.
3. Use your line to predict cone sales at 32°C.
4. Now use your line to predict cone sales at 0°C. Does the answer make physical sense? What does this tell you about extrapolation?
5. If every single data point lay exactly on a straight line, what would SSE equal, and why?

Work it out before reading on. x̄ = (20+25+30+35+40)/5 = 30. ȳ = (3+5+6+8+9)/5 = 31/5 = 6.2. The deviation products are (−10)(−3.2)=32, (−5)(−1.2)=6, (0)(−0.2)=0, (5)(1.8)=9, (10)(2.8)=28, summing to 75. The squared x-deviations are 100, 25, 0, 25, 100, summing to 250. So m = 75/250 = 0.3 and c = 6.2 − 0.3(30) = 6.2 − 9 = −2.8. At 32°C, predicted cones = 0.3(32) − 2.8 = 9.6 − 2.8 = 6.8. At 0°C, predicted cones = 0.3(0) − 2.8 = −2.8 — a negative number of cones, which is physically meaningless. This is extrapolation failing exactly the way the earlier section warned about: 0°C is far outside the 20–40°C range the data actually covered, and the straight-line pattern that fit nicely inside that range gives a nonsensical answer outside it. For question 5, if every point lies exactly on the line, every residual is 0, so every squared residual is 0, and SSE = 0 — the smallest value SSE can ever take.

Summary

Linear regression finds the single straight line, y = mx + c, that minimises the sum of squared residuals (SSE) across every point in your dataset, rather than a line that merely happens to touch a couple of convenient points. The slope formula, m = sum((x−x̄)(y−ȳ)) ÷ sum((x−x̄)²), captures how strongly x and y move together relative to how spread out x is; the intercept formula, c = ȳ − m · x̄, simply forces the line through the point of averages, a property every least-squares line satisfies without exception. Two structural guarantees follow directly from this construction and are worth remembering as sanity checks on your own hand calculations: the line always passes through (x̄, ȳ), and the residuals always sum to exactly zero. Once you have m and c, the line lets you predict y for a new x, but that prediction is only trustworthy for x-values reasonably close to the range your original data actually covered — stretching it further is extrapolation, and it can produce answers, like negative ice-cream sales, that no amount of correct arithmetic can rescue. When you translate this formula into code, the arithmetic will usually come out clean, exactly as it did for Meera's fares, but you should still compare floating-point results with a tolerance rather than exact equality, because not every dataset's numbers divide as neatly as hers did.