Accuracy and Error Metrics: Measuring ML Performance

The Spam Filter That Does Nothing — And Still Scores 96%

Suppose a small school in Pune runs its own email server, and someone builds a machine learning model to catch spam before it reaches teachers' inboxes. They collect 1,000 test emails to check how well the model works. Of these, only 40 are actually spam — the other 960 are genuine: homework submissions, circulars, a cricket practice schedule change. This 40-out-of-1,000 split is completely realistic; spam is the minority in most real inboxes.

Now imagine someone builds the laziest possible "model": it never flags anything. Every single email, spam or not, gets marked "not spam." If you check how often this lazy model is correct, here's what you get: it is right about all 960 genuine emails (it left them alone, correctly) and wrong about all 40 spam emails (it let every one through). That's 960 correct answers out of 1,000, which works out to 96%.

A model that does absolutely nothing useful — one that would let a phishing email through to a teacher without blinking — reports a 96% success rate. This is not a trick or a rare edge case. It is the single most common way people misjudge machine learning systems, and it is the reason this chapter exists. Before you can trust any number a model reports, you need to know exactly what that number is counting, and what it is hiding. That is what accuracy and error metrics are for.

Building the Confusion Matrix From Scratch

To see past a misleading percentage, you need to break a model's predictions into four honest categories instead of one vague "correct or wrong." Go back to the 1,000-email test set, but this time use a model that actually tries to detect spam — call it Model B. When you run Model B on all 1,000 emails and compare its predictions to the true labels, every single email falls into exactly one of four buckets:

• True Positive (TP): the email is really spam, and the model correctly said "spam." Model B gets 30 of these.
• False Negative (FN): the email is really spam, but the model said "not spam" and let it through. Model B gets 10 of these — spam that slipped past.
• False Positive (FP): the email is genuinely fine, but the model wrongly said "spam" and buried it in the spam folder. Model B gets 20 of these — a teacher's real email hidden from them.
• True Negative (TN): the email is genuinely fine, and the model correctly left it alone. Model B gets 940 of these.

Notice the pattern in the naming: the first word (True/False) tells you whether the model's prediction matched reality. The second word (Positive/Negative) tells you what the model predicted. A "false positive" is not "false" because it's a bad outcome in some vague sense — it's false because the model's positive prediction turned out to be incorrect. Arranging these four numbers into a 2×2 grid is called a confusion matrix, because it shows you exactly where and how the model gets "confused."

Check that the four numbers add up sensibly: TP + FN must equal the true number of spam emails, because every actual spam email is either caught (TP) or missed (FN). Here, 30 + 10 = 40 — correct. Similarly, FP + TN must equal the true number of genuine emails: 20 + 940 = 960 — also correct. And all four numbers together must equal the full test set: 30 + 10 + 20 + 940 = 1,000. This cross-check is worth doing every time; a confusion matrix that doesn't add up to your total sample size has an arithmetic error somewhere.

Accuracy: The Number Everyone Reaches For First

Accuracy is simply the fraction of all predictions that were correct:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

For Model B: Accuracy = (30 + 940) / 1000 = 970/1000 = 0.97, or 97%. That sounds excellent — until you remember the lazy "always say not-spam" model scored 96% on the exact same data by doing nothing at all. Model B only beat the do-nothing baseline by one percentage point, despite genuinely trying to detect spam. This is the core lesson of this chapter: accuracy is only a meaningful number when you also know the class balance of your data. On a perfectly balanced dataset (say, 500 spam and 500 not-spam), a lazy model can only guess right about half the time, so a 97% accuracy there really would be impressive. On this 4%-spam dataset, it barely clears the bar set by guessing.

A Second Model, and the Trade-Off Accuracy Hides

Now consider Model A — a much more aggressive spam filter, tuned to flag anything even slightly suspicious. Tested on the same 1,000 emails, Model A produces this confusion matrix: TP = 38 (it catches nearly all real spam), FN = 2 (only 2 spam emails slip through), FP = 200 (but it also wrongly flags 200 genuine emails as spam), and TN = 760.

Check the totals first: 38 + 2 = 40 actual spam (correct), 200 + 760 = 960 actual genuine mail (correct), and 38 + 2 + 200 + 760 = 1,000 overall (correct). Now compute Model A's accuracy: (38 + 760) / 1000 = 798/1000 = 79.8%.

Here is the trade-off in plain terms: Model A catches 38 out of 40 real spam emails — far better than Model B's 30 out of 40 — yet its overall accuracy is dramatically worse (79.8% versus 97%), because it buries 200 genuine emails in the spam folder along the way. If one of those 200 was a scholarship deadline notice or an exam hall-ticket link, that false positive could matter far more than a single missed spam email. A single accuracy number cannot tell you which kind of mistake a model is making, or which kind of mistake actually costs more. For that, you need to look inside the confusion matrix at precision and recall separately.

Precision and Recall: Two Different Questions

Precision and recall each ask a different, specific question about the model's mistakes, using only two of the four confusion-matrix numbers each.

Precision asks: "Of everything the model called spam, how much of it was actually spam?" It measures how much you can trust a positive prediction.

Precision = TP / (TP + FP)

Recall asks a completely different question: "Of everything that actually was spam, how much did the model catch?" It measures how thorough the model is at finding the positive cases.

Recall = TP / (TP + FN)

Compute both for each model. Model B: Precision = 30/(30+20) = 30/50 = 0.60 = 60%. Recall = 30/(30+10) = 30/40 = 0.75 = 75%. Model A: Precision = 38/(38+200) = 38/238 ≈ 0.1597 ≈ 16%. Recall = 38/(38+2) = 38/40 = 0.95 = 95%.

Now the two models' personalities are completely clear. Model A has very high recall (95%) — it is thorough, almost nothing slips past it — but terrible precision (16%) — the vast majority of what it labels "spam" is actually innocent mail, so a user would learn to distrust the spam folder entirely and start checking it manually, defeating the whole purpose of the filter. Model B has moderate precision (60%) and moderate recall (75%) — a more balanced, more usable trade-off. Neither "precision" nor "recall" alone tells the full story; you always need to look at the pair together, because a model can trivially maximise one of them at the expense of the other. A model that flags every email as spam gets 100% recall (it catches everything) with terrible precision. A model that never flags anything gets undefined-but-effectively-perfect precision (it makes zero false claims) with 0% recall.

Common misconception, corrected: many students assume "a model with higher accuracy is always the better model." Model A versus Model B proves this false. If your real priority is "never let a single spam email through, false alarms are an acceptable cost" (say, filtering out phishing attempts that try to steal bank OTPs), Model A's high recall might genuinely be the better choice despite its much lower accuracy. If your priority is "never hide a genuine email, occasional spam getting through is tolerable," Model B's balance may serve better. Accuracy alone cannot make this decision for you — it treats a missed spam email and a lost scholarship notice as identical mistakes, when in reality the costs are completely different, and only precision and recall (looked at separately) reveal that difference.

F1 Score: Combining Precision and Recall Into One Number

Sometimes you do want a single number to compare models quickly — for example, when automatically ranking twenty candidate models during training. The natural instinct is to average precision and recall. That instinct is wrong, and it's worth seeing exactly why with numbers.

Imagine an extreme model with Precision = 100% and Recall = 1%. A simple arithmetic average gives (1.00 + 0.01) / 2 = 0.505 = 50.5% — a number that looks like a coin-flip, "medium" model. But this model is nearly useless: it correctly identifies almost nothing (recall of 1%), even though whatever little it does flag happens to be correct. A single mediocre-sounding average badly disguises a genuinely broken model.

The F1 score fixes this by using the harmonic mean instead of the arithmetic mean:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

For the extreme example: F1 = 2 × (1.00 × 0.01) / (1.00 + 0.01) = 0.02 / 1.01 ≈ 0.0198 ≈ 2%. This correctly signals that the model is nearly worthless. The harmonic mean has a mathematical property the arithmetic mean lacks: it stays close to whichever of the two numbers is smaller. A model cannot hide a very weak precision or recall behind a strong one — both have to be reasonably good for F1 to be high.

Applying F1 to the spam models: Model B's F1 = 2 × (0.60 × 0.75) / (0.60 + 0.75) = 0.90 / 1.35 ≈ 0.667 ≈ 66.7%. Model A's F1 = 2 × (0.1597 × 0.95) / (0.1597 + 0.95) ≈ 0.3034 / 1.1097 ≈ 0.273 ≈ 27.3%. F1 now correctly ranks Model B as the stronger all-round model, something plain accuracy (97% vs 79.8%) also suggested, but which precision and recall alone (looked at separately) left ambiguous, since Model A "won" on recall.

One more related, simpler quantity worth knowing: error rate is just the flip side of accuracy — the fraction of predictions that were wrong: Error rate = 1 − Accuracy = (FP + FN) / Total. For Model B: 1 − 0.97 = 0.03 = 3%. It carries exactly the same information as accuracy, just phrased as "how often does it fail" instead of "how often does it succeed" — useful when you want to reason directly about the cost of mistakes.

When the Model Predicts a Number, Not a Category

Everything so far — confusion matrix, accuracy, precision, recall, F1 — applies to classification, where a model predicts a category like "spam" or "not spam." Many ML models instead predict a continuous number: tomorrow's maximum temperature, the price of a used car, the number of runs a batsman will score. This is called regression, and "was it exactly right or exactly wrong" no longer makes sense as a question — a predicted temperature of 37.9°C when the actual was 38°C is a tiny miss, not a total failure the way misclassifying spam as not-spam is. Regression needs its own error metrics that measure how far off a prediction was, not just whether it was right or wrong.

Suppose a model predicts Delhi's maximum temperature for five days. Define the error on each day as (actual − predicted):

Mean Absolute Error (MAE) averages the size of the errors, ignoring their sign (a model that's 2 degrees too high is treated the same as one that's 2 degrees too low):

MAE = (|2| + |-1| + |-4| + |2| + |0|) / 5
    = (2 + 1 + 4 + 2 + 0) / 5
    = 9 / 5 = 1.8°C

On average, this model's temperature prediction is off by 1.8°C. That's easy to explain to anyone — "typically about 2 degrees wrong." But MAE treats the day-3 miss (4 degrees off) as just "a bit worse" than the day-1 miss (2 degrees off), when in practice a 4-degree forecasting error might matter far more (imagine a farmer deciding whether to cover crops against a heatwave). Mean Squared Error (MSE) fixes this by squaring each error before averaging, which punishes large errors much more heavily than small ones:

MSE = (2² + (-1)² + (-4)² + 2² + 0²) / 5
    = (4 + 1 + 16 + 4 + 0) / 5
    = 25 / 5 = 5.0

Notice how the single 4-degree error (squared to 16) now dominates the sum, contributing nearly two-thirds of the total, while it only contributed 4 out of 9 (under half) to MAE. This is exactly the point of MSE: it flags models that make occasional large, dangerous errors, even if their typical error looks small. The drawback is that MSE's units are "degrees squared," which nobody can intuitively picture. Root Mean Squared Error (RMSE) fixes that by undoing the squaring at the end:

RMSE = √MSE = √5 ≈ 2.236°C

RMSE is back in the original unit (°C) like MAE, so it's directly comparable and interpretable, but because it's built from MSE, it still carries a built-in penalty for large errors — notice RMSE (2.24°C) is noticeably larger than MAE (1.8°C) precisely because of that one big 4-degree miss on day 3. When MAE and RMSE are close together, a model's errors are fairly uniform. When RMSE is much larger than MAE, it's a signal that a few large errors are hiding among many small ones — useful information a single accuracy-style number would never reveal.

Verifying the Formulas With Code

These formulas are simple enough to implement directly, which is also the best way to confirm you understand exactly what each one counts. Trace this function by hand before trusting its output:

def confusion_counts(actual, predicted):
    TP = TN = FP = FN = 0
    for a, p in zip(actual, predicted):
        if a == 1 and p == 1:
            TP += 1
        elif a == 0 and p == 0:
            TN += 1
        elif a == 0 and p == 1:
            FP += 1
        elif a == 1 and p == 0:
            FN += 1
    return TP, TN, FP, FN

actual    = [1, 0, 0, 1, 0]
predicted = [1, 0, 1, 1, 0]
TP, TN, FP, FN = confusion_counts(actual, predicted)
accuracy  = (TP + TN) / len(actual)
precision = TP / (TP + FP)
recall    = TP / (TP + FN)
print(TP, TN, FP, FN, accuracy, precision, recall)

Trace it pair by pair: (1,1)→TP, (0,0)→TN, (0,1)→FP, (1,1)→TP, (0,0)→TN. That gives TP=2, TN=2, FP=1, FN=0. So accuracy = 4/5 = 0.8, precision = 2/(2+1) ≈ 0.667, recall = 2/(2+0) = 1.0. The printed output is exactly 2 2 1 0 0.8 0.6666666666666666 1.0. Notice recall is a perfect 1.0 here — the model caught every actual positive case (both 1s in actual were predicted as 1) — while precision is lower because it also predicted one 1 that was actually a 0.

The regression metrics translate into code just as directly:

def error_metrics(actual, predicted):
    n = len(actual)
    errors = [a - p for a, p in zip(actual, predicted)]
    mae = sum(abs(e) for e in errors) / n
    mse = sum(e ** 2 for e in errors) / n
    rmse = mse ** 0.5
    return mae, mse, rmse

actual    = [38, 40, 35, 42, 37]
predicted = [36, 41, 39, 40, 37]
print(error_metrics(actual, predicted))

Trace: errors = [2, -1, -4, 2, 0]. mae = (2+1+4+2+0)/5 = 1.8. mse = (4+1+16+4+0)/5 = 5.0. rmse = 5.0 ** 0.5 ≈ 2.2360679... The function returns (1.8, 5.0, 2.23606797749979), matching the by-hand calculation from the temperature table exactly.

Choosing the Right Metric for the Job

There is no single "best" metric — the right choice depends on which kind of mistake is more expensive in your specific situation:

• A model screening for a serious disease should be judged mainly on recall: missing a true case (a false negative) can cost a life, while a false positive just means an extra confirmatory test. High recall matters even if precision suffers.
• A model recommending which court cases or loan applications to auto-approve should be judged mainly on precision: wrongly approving a bad case (a false positive) is far costlier than being cautious and asking a human to double-check a borderline one.
• A model like IRCTC predicting whether your waitlisted train ticket will get confirmed benefits from MAE or RMSE if it predicts a probability or a wait-list rank, since the size of the error (predicting rank 5 when the true rank was 40) matters, not just whether it was "close enough."
• When no single error type is clearly more costly, and you want one balanced number, F1 is the standard choice for classification, and RMSE is the standard choice for regression when you specifically want to penalise occasional large misses.

The habit worth building from this chapter is simple: whenever someone reports a single performance number for a machine learning model — especially "accuracy" — ask two follow-up questions before trusting it. First, how balanced are the classes in the test data (a 97% accuracy means something very different on a 50-50 dataset versus a 96-4 dataset)? Second, are false positives and false negatives equally costly here, or does one type of mistake matter far more than the other? A single number can never answer the second question; only a full confusion matrix, examined with precision, recall, and F1 (or MAE, MSE, and RMSE for regression), can.

Check Your Understanding

Q1. A hospital builds a model to screen for a rare condition that affects 2 out of every 1,000 patients. A "lazy" model always predicts "no condition." What accuracy does this lazy model get, and why is that number dangerously misleading?

Q2. Using Model A's confusion matrix from this chapter (TP = 38, FP = 200, FN = 2, TN = 760), calculate its precision and recall, and explain in one sentence what its behaviour is like for an actual user.

Q3. Why is F1 defined as the harmonic mean of precision and recall instead of their simple average? Support your answer with a numeric example.

Q4. A temperature-prediction model has these three daily errors (actual − predicted): +1°C, −5°C, +1°C. Compute its MAE and RMSE. Which metric is more affected by the −5°C error, and why?

Summary

• A confusion matrix splits every prediction into four honest categories: True Positive, False Negative, False Positive, and True Negative. Their sum must equal the total number of samples.
• Accuracy = (TP + TN) / Total. It is only meaningful when you also know the class balance — on imbalanced data, a model that predicts nothing but the majority class can score deceptively high while being useless.
• Precision = TP / (TP + FP) asks "how trustworthy are the model's positive predictions?" Recall = TP / (TP + FN) asks "how many of the real positives did the model actually catch?" They trade off against each other, and neither alone tells the full story.
• F1 = 2 × (Precision × Recall) / (Precision + Recall) is the harmonic mean of the two, and it correctly penalises a model that is very strong in one and very weak in the other — something a simple average would hide.
• Error rate = 1 − Accuracy; it carries the same information as accuracy, phrased as failure rate instead of success rate.
• For regression (predicting numbers, not categories), use MAE for a simple, evenly-weighted average error, MSE/RMSE when large errors should be penalised more heavily than small ones, and prefer RMSE over MSE when you want the result back in the original units.
• The right metric always depends on which kind of mistake costs more in your specific situation — recall matters most when missing a positive case is dangerous; precision matters most when a false alarm is expensive; F1 or RMSE give a single balanced number when neither error type clearly dominates.