Overfitting: Detecting and Solving the Problem

Two Students, One Exam Pattern

Every February, thousands of Class 9 students in India do the same thing: they pull out the last ten years of sample papers and go through them question by question. Consider two students doing this, Ananya and Rohan, both preparing for a Computer Science unit test on flowcharts.

Rohan takes a shortcut. He notices that three of the last five sample papers asked students to draw a flowchart for "finding the largest of three numbers," so he memorises that exact flowchart, symbol by symbol, arrow by arrow. He can reproduce it perfectly, without a single mistake, in under two minutes.

Ananya spends the same two hours differently. She works through six or seven different flowchart problems and asks, each time, why the decision box goes where it does, why the loop closes back to a particular point, what a diamond symbol means versus a rectangle. She is slower on any single problem than Rohan.

On the actual test day, the question is "draw a flowchart to find the largest of four numbers." Rohan freezes — his memorised diagram has exactly three input variables and he does not know how to extend it. Ananya adapts her understanding in under a minute and gets it right.

Rohan did not fail because he worked less hard. He scored perfectly on every practice paper he touched — arguably better than Ananya, who made small errors while practising. He failed because he optimised for the exact questions he had already seen instead of the pattern behind them. This is, almost exactly, what a machine learning model does when it overfits: it achieves an excellent score on the data it was trained on by memorising quirks specific to that data, and then performs badly on new data that follows the same general pattern but is not identical to anything it has seen.

What "Training" a Model Actually Means

When we build a machine learning model, we give it a training set — a collection of input-output examples — and let it adjust its internal parameters (numbers like slopes, weights, or thresholds) until its predictions match the outputs in that training set as closely as possible. The "closeness" is measured by an error (or loss): the difference between what the model predicted and what actually happened, usually squared and summed across all examples so that big misses count more than small ones.

The entire point of building the model, though, is never to predict the training examples — we already know those answers. The point is to predict correctly on new data the model has never seen: a new student's marks, a new patient's diagnosis, a new day's weather. Data set aside specifically for checking this is called a test set (or, when used during development to make decisions, a validation set).

Overfitting is what happens when a model's training error becomes very low — sometimes exactly zero — while its error on a test set stays high, or even gets worse, because the model has fit not just the underlying trend in the training data but also its random noise and one-off quirks, the way Rohan fit not the concept of "largest of n numbers" but the specific picture of one sample paper's answer.

The Class of Four — A Worked Example

Numbers make this precise. Suppose four students report how many hours they studied for a test and the marks (out of 100) they scored:

We want a model — a function that takes "hours studied" and predicts "marks." Two very different functions can both be fitted to this same table.

Fitting the Straight-Line Model

The simplest useful model is a straight line, y = a + bx. To fit it, we use the ordinary least-squares method you may already have used for two-variable statistics: find the mean of x and the mean of y, then compute the slope from how each point deviates from those means.

Mean of x: (1+2+3+4)/4 = 2.5. Mean of y: (40+45+50+58)/4 = 193/4 = 48.25.

Deviations of x from its mean: −1.5, −0.5, 0.5, 1.5. Deviations of y from its mean: −8.25, −3.25, 1.75, 9.75.

Slope b = Σ(x−x̄)(y−ȳ) / Σ(x−x̄)². The numerator is (−1.5)(−8.25) + (−0.5)(−3.25) + (0.5)(1.75) + (1.5)(9.75) = 12.375 + 1.625 + 0.875 + 14.625 = 29.5. The denominator is (−1.5)² + (−0.5)² + (0.5)² + (1.5)² = 2.25 + 0.25 + 0.25 + 2.25 = 5. So b = 29.5 / 5 = 5.9.

Intercept a = ȳ − b·x̄ = 48.25 − 5.9 × 2.5 = 48.25 − 14.75 = 33.5.

Our line is y = 33.5 + 5.9x. Checking its predictions against the real marks:

None of the predictions is exact. The sum of squared errors is 0.6² + 0.3² + 1.2² + 0.9² = 0.36 + 0.09 + 1.44 + 0.81 = 2.70. The line does not memorise the four students — it draws a general trend through them.

Fitting a Curve That Touches Every Point

Now here is a fact worth knowing precisely: given any four points with different x-values, there exists exactly one cubic curve — an equation with an x³ term — that passes through all four exactly, with zero error. Let's build it, using nothing beyond substitution.

Write the curve in a convenient form: y = a + b(x−1) + c(x−1)(x−2) + d(x−1)(x−2)(x−3). This form is convenient because each factor vanishes once x reaches the point it is "anchored" to, so substituting each of our four x-values solves for one unknown at a time.

Put x = 1: every bracket becomes 0, leaving y = a, so a = 40 (since y = 40 there).

Put x = 2: (x−1) = 1, the rest are 0, so y = a + b = 40 + b. Since y = 45, b = 5.

Put x = 3: (x−1)=2, (x−2)=1, last term still 0, so y = a + 2b + 2c = 40 + 10 + 2c = 50 + 2c. Since y = 50, 2c = 0, so c = 0.

Put x = 4: (x−1)=3, (x−2)=2, (x−3)=1, so y = a + 3b + 6c + 6d = 40 + 15 + 0 + 6d = 55 + 6d. Since y = 58, 6d = 3, so d = 0.5.

Our exact curve is y = 40 + 5(x−1) + 0.5(x−1)(x−2)(x−3). Multiplying this out into standard form (you can verify each step by expanding (x−1)(x−2)(x−3) = x³ − 6x² + 11x − 6 and collecting terms) gives y = 0.5x³ − 3x² + 10.5x + 32. Plug in x = 1, 2, 3, 4 and you will get exactly 40, 45, 50, 58 — zero error, every time. Its training error is 0, beating the line's 2.70 by a landslide.

If "lowest training error wins," the cubic has already won. But has it actually learned anything about how studying relates to marks, or has it just threaded a wire through four specific nails?

The Moment of Truth — A Fifth Student

A fifth student studied for 5 hours and scored 62 marks — data neither model was allowed to see while fitting. This is our test point.

The line predicts: 33.5 + 5.9 × 5 = 33.5 + 29.5 = 63.0. Against the real 62, that is off by exactly 1 mark.

The cubic predicts: 40 + 5(4) + 0.5(4)(3)(2) = 40 + 20 + 12 = 72. Against the real 62, that is off by 10 marks — ten times worse than the line, despite the cubic having had a perfect training score and the line having had a nonzero one.

Push it further. At 6 hours, the cubic gives 40 + 5(5) + 0.5(5)(4)(3) = 40 + 25 + 30 = 95. At 7 hours: 40 + 5(6) + 0.5(6)(5)(4) = 40 + 30 + 60 = 130 marks — out of 100. A model predicting a score above the maximum possible is not a subtle red flag; it is the model openly announcing that it has stopped describing reality and started reproducing the exact arithmetic of its four training points, wherever that arithmetic leads. The line, meanwhile, gives a believable 68.9 and 74.8 at 6 and 7 hours.

This is overfitting in its purest, most measurable form: a model with excellent — even perfect — performance on the data it was shown, and unreliable, sometimes absurd, performance the moment it meets anything new.

Seeing the Two Models Diverge

Underfitting — Failing for the Opposite Reason

Before going further, it helps to see the opposite mistake, because CBSE-style questions often ask you to distinguish the two. Suppose instead of any x-dependence at all, we used the laziest possible model: always predict the average, 48.25 marks, no matter how many hours anyone studied.

Its training errors are 40−48.25 = −8.25, 45−48.25 = −3.25, 50−48.25 = 1.75, and 58−48.25 = 9.75. Sum of squared errors = 68.06 + 10.56 + 3.06 + 95.06 ≈ 176.7 — far worse than even the straight line's 2.70, on the training data itself. This is underfitting: the model is too simple to capture even the pattern that is plainly there, so it performs badly on training data and on test data alike. Overfitting and underfitting are not two names for "bad model" — they are opposite failure directions, and the fix for one (add complexity) is often the cause of the other if pushed too far.

Formal Definitions You Need for Exams

With the worked example behind us, the vocabulary now has something concrete to attach to:

• Training error: how wrong the model is on the exact data it learned from. (Line: 2.70. Cubic: 0. Average-only model: 176.7.)
• Test error / validation error: how wrong the model is on data it never saw during training. (Line: off by 1 mark. Cubic: off by 10 marks.)
• Generalisation gap: the difference between test error and training error. A small gap means the model's training performance is a trustworthy preview of real-world performance. A large gap — training error near zero, test error large — is the fingerprint of overfitting.
• Model capacity / complexity: roughly, how many independently adjustable numbers (parameters) the model has relative to how much data it was given. Our line has 2 parameters (a, b) for 4 data points. Our cubic has 4 parameters (equivalent to a, b, c, d) for 4 data points — exactly enough to force a perfect fit, which is precisely why it had no error left over to be "wrong" about noise, and instead absorbed the noise into itself.

How to Actually Detect Overfitting

In practice you rarely have the luxury of a dramatic 130-marks-out-of-100 warning sign. The reliable method is structural, not visual: before training begins, split your available data into a training set and a validation set (a common split is 80% training, 20% validation), fit the model only on the training portion, and then check its error on the validation portion, which it never touched. Watch two numbers as you make the model more complex — more polynomial terms, more decision-tree depth, more neural network layers:

Training error keeps falling — it almost always can, if you let the model grow complex enough, exactly as our cubic drove it to zero. Validation error falls at first too, because a model that is too simple is bad at both; but past some point it turns and starts rising even as training error keeps dropping. That turning point is the "sweet spot," and everything to its right — low training error, rising validation error, a widening gap between the two lines — is the operational definition of overfitting. This curve, not a single number, is what you actually plot and read in practice.

You can confirm the same story numerically. Fitting both models from our worked example in Python and checking them against the unseen fifth student:

import numpy as np

hours = np.array([1, 2, 3, 4])
marks = np.array([40, 45, 50, 58])

line_coeffs  = np.polyfit(hours, marks, 1)   # 2 parameters
cubic_coeffs = np.polyfit(hours, marks, 3)   # 4 parameters, exact fit

line_model  = np.poly1d(line_coeffs)
cubic_model = np.poly1d(cubic_coeffs)

test_hours, actual_marks = 5, 62

print("Line predicts: ", round(line_model(test_hours), 1))   # 63.0
print("Cubic predicts:", round(cubic_model(test_hours), 1))  # 72.0
print("Actual marks:  ", actual_marks)                       # 62

(Tiny rounding differences of a few thousandths may appear because polyfit solves the system numerically rather than by the hand substitution we did above, but the values will match to one decimal place.) When a validation split is not available because the dataset is very small, a more thorough version of the same idea is k-fold cross-validation: split the data into, say, 5 equal chunks, train five separate times using 4 chunks each time and validate on the 5th (a different chunk each round), then average the five validation errors. This uses every data point for both training and validation at some point, giving a more reliable estimate than a single fixed split when data is scarce.

A Misconception Worth Correcting

A very common belief is: "A model that fits the training data more closely is a better model." Our worked example is a direct counter-example — the cubic fit the training data perfectly (error 0, unbeatable by that measure) and was the worse model, ten times worse at the one job that mattered: predicting a real, unseen student's marks. Training accuracy tells you how well a model memorised; only validation or test accuracy tells you whether it learned. Never trust a reported accuracy figure that does not specify which set it was measured on.

A second, related misconception is that overfitting is a problem unique to deep neural networks with millions of parameters. It is not — our overfitting example used ordinary polynomial regression with just four numbers to adjust, the kind of thing you could compute by hand, as we just did. Overfitting is a relationship between model capacity and data quantity, not a property of any one algorithm. Decision trees overfit by growing branches for individual noisy examples; k-nearest-neighbours overfits when k is too small; even a linear regression overfits if you feed it more features than data points.

Five Ways to Fix Overfitting

1. Get more training data. A cubic can bend through 4 points exactly, but it cannot bend through 40 points that all follow a roughly linear trend without incurring real error somewhere — extra genuine data points act as constraints that prevent the curve from swinging freely between them.
2. Reduce model complexity. Choosing a lower polynomial degree, a shallower decision tree, or fewer input features directly reduces how much freedom the model has to chase noise. This is the most direct fix for our worked example: use the line, not the cubic, when you have only 4 points.
3. Regularisation. Instead of only minimising training error, add a penalty for having large coefficients, so the model has to "earn" complexity rather than get it for free. Expand our cubic into standard form and it reads y = 0.5x³ − 3x² + 10.5x + 32. If we penalise the sum of squared coefficients (excluding the constant) with a weight λ = 0.5: the line's penalty is 0.5 × 5.9² = 0.5 × 34.81 ≈ 17.4, giving a regularised score of 2.70 + 17.4 ≈ 20.1. The cubic's penalty is 0.5 × (0.5² + 3² + 10.5²) = 0.5 × (0.25 + 9 + 110.25) = 0.5 × 119.5 ≈ 59.8, giving a regularised score of 0 + 59.8 = 59.8. Even though the cubic's raw training error was zero, its regularised score is nearly three times worse than the line's — so a model selected by minimising this combined score would correctly choose the line. This is exactly how ridge regression decides against overly wiggly fits.
4. Early stopping. For models trained iteratively (gradient descent, neural networks), track validation error after every training round and stop the moment it starts climbing again — the same "sweet spot" marked in the complexity-vs-error diagram above, except tracked over training time instead of model size.
5. Pruning and dropout. For decision trees, pruning removes branches that were grown to fit a handful of specific training examples rather than a general rule, checked by seeing whether removing them improves validation accuracy. For neural networks, dropout randomly disables a fraction of neurons during each training step, which prevents the network from becoming overly reliant on any one memorised pathway.

Quick Recap

• Overfitting: training error very low (even zero), test/validation error high — the model memorised training-specific noise instead of learning the true pattern.
• Underfitting: both training and test error are high — the model is too simple to capture even the real pattern.
• The reliable detection tool is a held-out validation set (or k-fold cross-validation), plotted as training error vs validation error across increasing model complexity; the point where validation error starts rising again is the target complexity.
• Fixes: more data, less complexity, regularisation (penalise large coefficients), early stopping, and technique-specific tools like tree pruning or dropout.
• A perfect training score is not proof of a good model — verify against data the model never trained on.

Check Your Understanding

1. Using the line y = 33.5 + 5.9x from our worked example, what marks would it predict for a student who studied 6 hours? Compare this to the cubic's prediction of 95 for the same student. Which is more believable, and why?
2. Explain in one sentence why a model predicting 130 marks out of a possible 100 is a reliable warning sign of overfitting, not just an unlucky guess.
3. A classmate says, "My model got 99% accuracy, so it must be excellent." What is the one question you must ask before agreeing, and why does our worked example show it matters?
4. A decision tree gets 60% accuracy on its training data and 58% on its validation data. Is this closer to overfitting or underfitting? Justify your answer using the generalisation-gap idea.
5. If our class-of-four dataset had 40 students instead of 4, explain why fitting an exact cubic through all of them would become impossible, and why this connects to "more data reduces overfitting."
6. In the regularisation example, recompute the regularised score for the line and the cubic using λ = 1 instead of λ = 0.5. Does the line still win?

Think About It

Think about this: How would you explain overfitting: detecting and solving the problem to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.