Training and Testing: Why You Can't Grade Your Own Exam

The Student Who "Solved" Every Practice Paper

Before your Class 9 pre-boards, imagine a classmate — call her Ananya — who takes the last five years of CBSE Science previous-year question papers and does something clever, or so it seems. She solves each paper, checks the answer key, and where she gets a question wrong, she doesn't relearn the concept — she simply memorises that exact question's exact answer. She repeats this until she can reproduce all 100 questions from those five papers with 100% accuracy. She feels ready.

Then the actual pre-board arrives. It is written by a different teacher, testing the same syllabus, the same concepts — but with new numbers in the numericals, new phrasing in the definitions, new combinations of ideas in the assertion-reason questions. Ananya scores 61%. She is confused: "But I got every single practice question right!"

Nothing mysterious happened here. Ananya never learned photosynthesis or Newton's laws — she learned "question 14 on the 2021 paper has the answer 'mitochondria'." Her perfect score on the practice papers measured her memory, not her understanding. The pre-board, built from questions she had never seen, measured the thing that actually mattered.

This chapter is about the machine-learning version of exactly this problem. Every time we build a model that learns a pattern from data, we face the same question examiners face: did it actually learn the concept, or did it just memorise the answer key? And just as no teacher would let Ananya grade her own paper using the same key she memorised from, no machine learning practitioner should ever judge a model using the same data it learned from. That is the single idea this entire chapter builds toward.

Training Set and Test Set: The Two Piles of Data

In machine learning, before you do anything else, you split your labelled data — examples together with the correct answers — into two separate piles that never mix:

• Training set: the data the model is allowed to look at while it is learning. This is Ananya's five years of previous-year papers with the answer key open in front of her.
• Test set: data that is set aside, locked away, and never shown to the model until training is completely finished. This is the pre-board exam. The model's performance here is the only honest measurement of whether it actually learned something useful.

The rule is strict on purpose: once you start letting the "learning" process see the test data, even a little, even by accident, the test set stops being a fair exam and becomes just another practice paper. We will come back to exactly how this accidental leakage happens later in the chapter, because it is one of the most common real mistakes beginners make.

A Worked Example: Predicting Pass or Fail from Hours Studied

Let's make this concrete with numbers you can trace by hand. Suppose a school collects data from 16 Class 9 students: for each student, how many hours per week they studied for a unit test, and whether they actually got a Pass or a Fail. We want to build a model that predicts Pass/Fail from hours studied alone.

We split the 16 students into two piles before we do anything else: 11 students go into the training set (the model is allowed to study these, answer key included), and 5 students are held back, untouched, as the test set — the pre-board these 11 students' worth of "learning" will eventually be judged against.

Here is the actual data, sorted by hours studied so the pattern is easier to see. The training set (11 students):

• 1 hour studied → Fail
• 2 hours studied → Fail
• 3 hours studied → Fail
• 4 hours studied → Fail
• 5 hours studied → Pass (this student already knew the material well; low hours, still passed)
• 6 hours studied → Pass
• 7 hours studied → Fail (this student studied inefficiently — hours alone didn't save them)
• 8 hours studied → Pass
• 9 hours studied → Pass
• 10 hours studied → Pass
• 11 hours studied → Pass

Notice this is not a perfectly clean pattern — the 5-hour student passed and the 7-hour student failed, breaking a simple "more hours always means Pass" rule. Real data is messy like this, and that messiness is exactly what makes the training-versus-testing question interesting. Counting up: 6 students passed, 5 failed.

The test set (5 students, held back and never used for learning):

• 2 hours studied → Fail
• 5 hours studied → Fail
• 8 hours studied → Pass
• 9 hours studied → Pass
• 12 hours studied → Pass

Three Different Ways to "Learn" the Same Data

Now here is the key experiment. We will build three different models from the training set above, and see how each one behaves — both on the training set it learned from, and on the test set it has never seen. Think of these as three different students revising for the same pre-board:

• The Memorizer. This model stores every single (hours, result) pair it sees during training, like a lookup table — a dictionary. Ask it about a value of hours it has already seen, and it recites the memorised label back perfectly. Ask it about a value it has never seen, and it has nothing to fall back on — it just guesses. This is Ananya, memorising question-answer pairs instead of concepts.
• The Constant Guesser. This model doesn't even look at the hours. It notices that "Pass" was the more common result in training (6 out of 11) and simply predicts "Pass" for every single student, no matter what. This is the student who leaves every answer blank except writing the most statistically likely option, having learned nothing about the actual subject.
• The Threshold Rule. This model looks for one genuine, simple pattern: "students who study 6 or more hours a week tend to pass; students who study less tend to fail." It applies this one rule — hours ≥ 6 → Pass, otherwise Fail — to every student, seen or unseen. This is the student who actually studied the concept of the relationship between effort and outcome, not the specific past papers.

Tracing the Code, Line by Line

Here is all three models written as plain Python, along with the training and test data exactly as listed above:

train = [
    (1, "Fail"), (2, "Fail"), (3, "Fail"), (4, "Fail"),
    (5, "Pass"), (6, "Pass"), (7, "Fail"), (8, "Pass"),
    (9, "Pass"), (10, "Pass"), (11, "Pass"),
]

test = [
    (2, "Fail"), (5, "Fail"), (8, "Pass"),
    (9, "Pass"), (12, "Pass"),
]

def accuracy(predict, data):
    correct = 0
    for hours, actual in data:
        if predict(hours) == actual:
            correct += 1
    return round(100 * correct / len(data), 1)

# Model 1: The Memorizer
memory = dict(train)
def memorizer(hours):
    if hours in memory:
        return memory[hours]
    return "Fail"          # never seen this value — just guesses

# Model 2: The Constant Guesser
def constant_guesser(hours):
    return "Pass"           # "Pass" was the majority label in training

# Model 3: The Threshold Rule
def threshold_rule(hours):
    if hours >= 6:
        return "Pass"
    return "Fail"

models = [
    ("Memorizer", memorizer),
    ("Constant Guesser", constant_guesser),
    ("Threshold Rule", threshold_rule),
]

for name, model in models:
    tr = accuracy(model, train)
    te = accuracy(model, test)
    print(name, "-> train:", tr, " test:", te)

Before running this in your head, notice the important design detail: memory = dict(train) builds the Memorizer's lookup table only from the training list. The test list is never shown to any of the three functions while they are being built — exactly like the pre-board being sealed until exam day.

Let's hand-trace each model's training accuracy first, since dict(train) uses the exact same 11 pairs it will be tested against here.

Memorizer on training data: every one of the 11 training pairs is already sitting in memory, so every lookup returns exactly the label that was stored — 11 out of 11 correct. That's 100 * 11 / 11 = 100.0.

Constant Guesser on training data: it predicts "Pass" for all 11 students. It's right exactly when the actual label was Pass — that happened for the students at 5, 6, 8, 9, 10, and 11 hours: 6 out of 11 correct. That's 100 * 6 / 11 = 54.545..., which rounds to 54.5.

Threshold Rule on training data: apply "hours ≥ 6 → Pass" to all 11 rows:

• 1, 2, 3, 4 hours (all actual Fail) → predicted Fail. Correct × 4.
• 5 hours (actual Pass) → predicted Fail (since 5 < 6). Wrong.
• 6 hours (actual Pass) → predicted Pass. Correct.
• 7 hours (actual Fail) → predicted Pass (since 7 ≥ 6). Wrong.
• 8, 9, 10, 11 hours (all actual Pass) → predicted Pass. Correct × 4.

That's 9 correct out of 11: 100 * 9 / 11 = 81.818..., rounding to 81.8. Unlike the Memorizer, the Threshold Rule doesn't get a perfect score on its own training data — it makes two honest mistakes, on exactly the two "exception" students who broke the simple pattern.

Now the moment that matters — testing on the 5 students none of the three models has ever seen:

Memorizer on test data: check each test value against the memorised dictionary from training, which contains the keys 1 through 11.

• 2 hours: was in training (labelled Fail). Memorizer recalls Fail. Actual is Fail. Correct.
• 5 hours: was in training — but labelled Pass there (a different student, same hour value). Memorizer recalls Pass. Actual test label is Fail. Wrong.
• 8 hours: was in training (Pass). Memorizer recalls Pass. Actual is Pass. Correct.
• 9 hours: was in training (Pass). Memorizer recalls Pass. Actual is Pass. Correct.
• 12 hours: never appeared in training at all — training only went up to 11. Memorizer has no memory of it, falls back to its default guess, "Fail". Actual is Pass. Wrong.

3 out of 5 correct: 100 * 3 / 5 = 60.0.

Constant Guesser on test data: predicts "Pass" for all 5. It's correct wherever the actual label is Pass: that's the 8, 9, and 12-hour students — 3 out of 5. Also 60.0.

Threshold Rule on test data: apply hours ≥ 6 → Pass:

• 2 hours (Fail) → predicted Fail. Correct.
• 5 hours (Fail) → predicted Fail (5 < 6). Correct.
• 8 hours (Pass) → predicted Pass. Correct.
• 9 hours (Pass) → predicted Pass. Correct.
• 12 hours (Pass) → predicted Pass. Correct.

All 5 correct: 100 * 5 / 5 = 100.0.

Running the actual program prints exactly what we just computed by hand:

Memorizer -> train: 100.0  test: 60.0
Constant Guesser -> train: 54.5  test: 60.0
Threshold Rule -> train: 81.8  test: 100.0

Reading the Results: What the Numbers Are Actually Telling You

Look at the Memorizer's row first: 100.0 on training, 60.0 on the test. That gap — a huge score on data it already knew the answers to, and a much smaller score on data it had never seen — is called overfitting. The model didn't learn a rule about hours and results; it learned a lookup table of specific values. On the training set, a lookup table is unbeatable, because every question is one it has already seen the answer to. On the test set, a lookup table is nearly useless the moment it meets a value it hasn't memorised — like the 12-hour student, where it just defaulted to a guess.

Now look at the Constant Guesser: 54.5 on training, 60.0 on test. It isn't overfitting — its two scores are close to each other — but both scores are mediocre, because it never bothered to look at the hours at all. This is called underfitting: the model is too simple to capture the real relationship in the data, so it performs poorly everywhere, train and test alike.

Now look at the Threshold Rule: 81.8 on training, 100.0 on test. This model made two honest mistakes during training — it didn't memorise the two "exception" students, it looked for the general pattern and accepted that a couple of data points wouldn't fit it perfectly. On this particular test set, that general pattern happened to classify every one of the 5 new students correctly. The training score being lower than the Memorizer's is not a weakness here — it is the sign of a model that refused to contort itself around noise.

A common misconception, corrected directly: many beginners assume "higher training accuracy always means a better model." The Memorizer proves this false in the most direct way possible — it scored a perfect 100.0 on training, the highest of all three models, yet it was the worst model on the test set, tied with the Constant Guesser that never even looked at the data properly. Training accuracy tells you how well a model can reproduce answers it has already seen. It tells you nothing, by itself, about how the model will behave on a new student, a new transaction, a new photograph — anything it hasn't already memorised. Only test accuracy, measured on data the model never touched during learning, estimates that.

One caution worth stating plainly, especially with a dataset this small: 5 test students is a tiny sample, and the Threshold Rule scoring higher on test (100.0) than on train (81.8) here is partly a small-sample coincidence — with more test data its score would likely settle somewhere closer to its training accuracy. The lesson to take away isn't "test accuracy above train accuracy is the goal." It's this: a large gap where training accuracy is far higher than test accuracy is the warning sign of overfitting, and the way you catch that warning sign at all is by insisting on evaluating with data the model never learned from.

The Golden Rule: Never Let the Model Study the Test

Here is a trap even careful beginners fall into. Suppose you're not satisfied with the Threshold Rule's cutoff of 6 hours — maybe 5 hours, or 7, would do better. So you try threshold 5 on the test set, then threshold 6, then threshold 7, and you keep the one that scores highest on the test set. It feels harmless — you're not changing the training data. But you are quietly turning the test set into a second training set: you are choosing your final model because of how it performs on data that is supposed to be an untouched, honest exam. Do this enough times, trying enough variations, and you will eventually find some rule that scores well on your 5 test students purely by chance — without it being any better at the real underlying pattern. This is called data leakage, and it is exactly as dishonest as Ananya somehow getting an advance look at the pre-board paper before deciding how to revise.

The fix used throughout real machine learning work is a three-way split. Alongside the training set and test set, you carve out a third pile — a validation set — used specifically for trying out choices like "should the threshold be 5, 6, or 7?" You tune and compare freely using the validation set, exactly as many times as you like, because it was never claimed to be the final honest exam. Only once you've picked your best setting do you touch the test set — a single time, at the very end — to report the number that actually matters. In our example, this would mean carving a few students out of the 11-student training set purely for trying different thresholds, leaving the 5-student test set sealed until the final threshold rule is locked in.

Why This Matters Beyond the Classroom

This exact discipline is what stands between a machine learning system that works and one that quietly fails the moment it meets the real world. Consider a bank or a UPI payment app building a system to flag suspicious transactions as fraud. If the fraud-detection model is only ever checked against the same historical transactions it was trained on, it can look outstanding on paper — a Memorizer that has simply learned to recognise specific past transaction patterns — while missing every genuinely new fraud technique that shows up next month, because those weren't in its "answer key." Only by holding back a separate, untouched batch of past transactions as a test set — and only trusting the number that comes from that untouched batch — can a team honestly estimate how the system will behave on tomorrow's transactions, which is the only thing that actually matters once it's deployed.

If you go on to take Artificial Intelligence as a skill subject in CBSE, you will meet this same idea again, formally, as the "Data Acquisition" and "Modelling" stages of the AI Project Cycle — and the instruction to never evaluate a model on the data it was trained on will still be the very first rule you're taught.

Check Your Understanding

A gardener records how many days a tomato plant went without water, and whether it ended up Healthy or Wilted. The rule being tested is: "Wilted if days without water ≥ 4, otherwise Healthy."

Training data (6 plants): 1 day→Healthy, 2 days→Healthy, 3 days→Healthy, 4 days→Wilted, 5 days→Healthy (an unusually hardy plant), 6 days→Wilted.

Test data (3 plants, never used above): 2 days→Healthy, 4 days→Wilted, 7 days→Wilted.

1. Apply the rule to all 6 training plants and compute the training accuracy as a percentage.
2. Apply the same rule to all 3 test plants and compute the test accuracy.
3. Based on the two accuracies, is this rule closer to overfitting, underfitting, or generalising well? Justify your answer using the numbers.
4. A "Memorizer" model is trained on the same 6 plants (as a days→label lookup table). What would it predict for a plant that went 10 days without water, and why is that a problem?
5. Why must the 3-plant test set be checked only once, at the very end, rather than after every small change to the rule?

Worked answers:

1. Checking each training plant: 1, 2, 3 days (<4) → predicted Healthy, all actually Healthy — correct ×3. 4 days → predicted Wilted, actually Wilted — correct. 5 days → predicted Wilted (since 5 ≥ 4), but actually Healthy — wrong. 6 days → predicted Wilted, actually Wilted — correct. That's 5 correct out of 6: 100 × 5 / 6 = 83.3%.

2. 2 days → predicted Healthy, actual Healthy, correct. 4 days → predicted Wilted, actual Wilted, correct. 7 days → predicted Wilted, actual Wilted, correct. All 3 correct: 100%.

3. Generalising well. The rule made one honest mistake on training (the 5-day hardy-plant exception) rather than distorting itself to fit that one unusual case, and its test accuracy (100%) is not lower than its training accuracy (83.3%) — there is no large train-high, test-low gap, which is the signature of overfitting. A rule this simple, scoring well on unseen data, is a sign it captured something real about water stress rather than memorising quirks of six specific plants.

4. The Memorizer's lookup table only has entries for 1, 2, 3, 4, 5, and 6 days — it has never seen the value 10 in training, so it has no memorised answer to return. It would have to fall back on some arbitrary default guess, completely unrelated to whether 10 days without water is actually dangerous for a tomato plant. This is precisely the weakness that made the Memorizer fail on the 12-hour student earlier in the chapter — it cannot generalise to any input value it did not literally see during training, no matter how obviously the pattern should extend.

5. Checking the test set repeatedly and adjusting the rule based on what you see turns the test set into a second training set by proxy — you begin fitting your choices to it, exactly as discussed under data leakage. Once that happens, its score is no longer an honest estimate of how the rule performs on truly new plants; it only tells you how well you've tuned the rule to those specific 3 test plants. Checking it exactly once, after every decision has already been finalised, is what keeps it a fair, final exam.

Summary

• A model's training set is the data it is allowed to learn from; its test set is separate, held-back data used only to measure how well it generalises — never used during learning.
• Training accuracy measures how well a model reproduces answers on data it has already seen; on its own, it says nothing about performance on new data. A model that has simply memorised its training data (a "Memorizer") can score perfectly here while performing poorly elsewhere.
• Test accuracy, measured on data the model never touched while learning, is the honest estimate of real-world performance — the equivalent of the actual exam, not the practice papers.
• Overfitting is a large gap where training accuracy is much higher than test accuracy — the model memorised noise and specifics instead of the underlying pattern. Underfitting is when both training and test accuracy are mediocre — the model is too simple to capture the real pattern at all. A model that generalises well keeps the gap between the two small while both stay reasonably high.
• Data leakage happens when the test set is checked repeatedly and used, even indirectly, to guide decisions about the model — which quietly destroys its value as an honest, final exam. The fix is a validation set, carved separately from training data, used freely for tuning choices like a threshold value, while the test set is touched exactly once, at the very end.