Confusion Matrix: Evaluating Classification Models

Why "95% Accurate" Can Be a Lie

Your phone buzzes. A message has just arrived: "Congratulations! You have won ₹10,000 cashback. Click here to claim before it expires." Somewhere between the sender's network and your inbox, an AI model looked at that message and made a decision in a few milliseconds: spam or not spam. Every telecom operator and email provider in India runs a model exactly like this in the background, sorting millions of SMS messages and emails a day — a task that Indian telecom regulations now explicitly require operators to handle using AI/ML-based filtering systems, precisely because manually checking every message is impossible at that scale.

Now suppose the company that built this filter tells you, "Our model is 95% accurate." Sounds impressive. But here is the question a good data scientist — and a good CBSE Computer Science student — should immediately ask back: accurate at what, exactly? A single accuracy number hides a huge amount of information. It does not tell you whether the model's mistakes are mostly letting real spam through into your inbox, or mostly blocking your bank's One-Time Password (OTP) message and calling it spam. Those two kinds of mistakes have very different consequences, and "95% accurate" treats them as if they were the same.

This is exactly the gap that a confusion matrix is built to close. It does not report a single number. It reports a small table that shows every possible way a classifier's prediction can match or mismatch reality — which turns a vague claim like "95% accurate" into an honest, checkable picture of what the model actually gets right and wrong.

Two Different Ways to Be Wrong

Think about the spam filter's job on any single message. There are exactly two things that can be true about the message (it really is spam, or it really is not), and exactly two things the model can predict (spam, or not spam). That gives four possible combinations, and it is worth walking through each one slowly, because the whole chapter is built on this idea.

1. The message really is spam, and the model correctly predicts spam. A win — the filter did its job. In evaluation language this is called a True Positive (TP): the model predicted the "positive" class (here, "spam" is the class we care about detecting), and it was true.
2. The message really is not spam, and the model correctly predicts not spam. Also a win — a genuine message (say, your school's fee-reminder SMS) reaches you untouched. This is a True Negative (TN): the model predicted the "negative" class, and it was true.
3. The message really is not spam, but the model wrongly predicts spam. This is a False Positive (FP) — the model raised a false alarm. If this happens to your bank's OTP message, you might miss a UPI transaction deadline because the message got silently filtered out. False positives are false alarms.
4. The message really is spam, but the model wrongly predicts not spam. This is a False Negative (FN) — the model missed a real case. The spam message lands in your inbox as if it were legitimate. False negatives are missed detections.

A useful trick for remembering the names: the second word (Positive/Negative) tells you what the model predicted. The first word (True/False) tells you whether that prediction was correct. "False Positive" therefore literally means "the model said positive, and it was wrong to." This naming pattern generalizes to every binary classifier you will ever evaluate — a medical test predicting "disease present," a bank system predicting "fraudulent transaction," or an admissions model predicting "will pass."

Building the Confusion Matrix

A confusion matrix simply arranges these four counts into a 2×2 grid: one axis for what actually happened (the true label), and one axis for what the model predicted. Every one of your test messages falls into exactly one of the four cells, so the four counts always add up to your total test size. Here is the matrix filled in for a batch of 200 real SMS messages that a telecom company set aside to test its spam filter — 50 of which were genuinely spam and 150 of which were genuinely legitimate messages.

Read the matrix the way you would read the story of the model's test run. The model looked at 50 truly spam messages and correctly flagged 42 of them (TP), but let 8 slip through as if they were genuine (FN). It also looked at 150 truly genuine messages and correctly left 135 of them alone (TN), but wrongly flagged 15 of them as spam (FP). Notice how the rows add up to the real-world totals: the "Actual: Spam" row is 42 + 8 = 50, and the "Actual: Not Spam" row is 15 + 135 = 150 — exactly the 50 and 150 the telecom company started with. That row-sum check is a good habit: if your rows don't add up to how many real spam and real non-spam messages you actually tested, you have made an arithmetic mistake somewhere in building the matrix.

From Counts to Metrics: Accuracy, Precision, Recall

The four raw counts (TP, TN, FP, FN) are the foundation, but people rarely quote them directly — they combine them into ratios that are easier to compare across models. Three matter most, and each answers a genuinely different question.

Accuracy answers: "out of everything, what fraction did the model get right?" It combines both correct outcomes over the whole test set:

Accuracy = (TP + TN) / (TP + TN + FP + FN)
         = (42 + 135) / 200
         = 177 / 200
         = 0.885  →  88.5%

Precision answers a narrower question: "out of every message the model called spam, how many actually were spam?" It only looks at the model's positive predictions and checks how trustworthy they are:

Precision = TP / (TP + FP)
          = 42 / (42 + 15)
          = 42 / 57
          ≈ 0.737  →  73.7%

Precision matters when a false alarm is expensive. If your bank's OTP gets misclassified as spam (a false positive) and you miss a payment window, that is exactly the cost precision is measuring. Low precision means the model cries "spam" too often on innocent messages.

Recall (also called sensitivity) answers yet another question: "out of every message that was truly spam, how many did the model actually catch?" It only looks at the real positive cases and checks how many the model found:

Recall = TP / (TP + FN)
       = 42 / (42 + 8)
       = 42 / 50
       = 0.84  →  84%

Recall matters when missing a real case is expensive — a real fraud transaction, a real disease, a real spam message carrying a phishing link. Low recall means real danger is slipping through undetected.

There is a fourth ratio worth knowing, specificity, the mirror image of recall on the negative side: "out of every message that was truly not spam, how many did the model correctly leave alone?"

Specificity = TN / (TN + FP)
            = 135 / (135 + 15)
            = 135 / 150
            = 0.90  →  90%

Notice that all four numbers came from the exact same 2×2 table — you are not re-testing the model, just asking the four counts different questions. This is precisely why the confusion matrix, not accuracy alone, is the real starting point of evaluation: it is the one artifact from which every other metric can be derived.

One metric sometimes used to combine precision and recall into a single number is the F1 score, their harmonic mean:

F1 = 2 × (Precision × Recall) / (Precision + Recall)
   = 2 × (0.737 × 0.84) / (0.737 + 0.84)
   ≈ 1.238 / 1.577
   ≈ 0.785  →  78.5%

F1 is useful exactly when you cannot decide whether precision or recall matters more and want one balanced number — but remember it is a compromise, and a compromise can hide the fact that one of the two underlying numbers is genuinely poor.

The Accuracy Trap: A Common Misconception

Here is a mistake even confident students make: assuming that a model reporting 99% accuracy must be an excellent model, no matter what it is predicting. This is false, and the confusion matrix is exactly the tool that exposes why.

Imagine a bank builds an AI system to flag fraudulent UPI transactions. Suppose, as a test, they run it on 10,000 transactions, of which only 50 are genuinely fraudulent (fraud is rare — that is realistic; most transactions are legitimate). Now imagine a lazy "model" that has learned nothing at all and simply predicts "not fraud" for every single transaction, no matter what. Its confusion matrix looks like this:

• TP (correctly caught fraud) = 0
• FN (fraud it missed) = 50
• FP (legitimate transactions wrongly blocked) = 0
• TN (legitimate transactions correctly left alone) = 9,950

Compute its accuracy:

Accuracy = (TP + TN) / Total = (0 + 9950) / 10000 = 0.995  →  99.5%

Ninety-nine point five percent accuracy — and the model has never once caught a single fraudulent transaction. Its recall is 0/(0+50) = 0%, which is the truer picture of how useless it is for the one job it was built to do. Its specificity is a perfect 9950/9950 = 100%, which sounds great until you realize it is trivially easy to get: a model that never predicts fraud will never wrongly accuse an innocent transaction either. This is called the accuracy paradox: when the classes are imbalanced — one class (legitimate transactions) vastly outnumbers the other (fraud) — accuracy can stay high even while the model completely fails at the rare, important class. This is precisely why a confusion matrix, which exposes recall and precision separately, is considered far more trustworthy than a single accuracy figure for problems like fraud detection, disease screening, or spam filtering, where the interesting class is usually the smaller one.

The correction to carry forward: always ask what fraction of the dataset the positive class actually makes up before trusting an accuracy number. If positives are rare, check recall and precision directly instead.

Computing a Confusion Matrix in Python

You rarely count TP/FP/FN/TN by hand for a real dataset — you let code do it. Here is a small worked example with ten SMS messages, tracing exactly what the program does and what it prints.

# actual[i] is the true label of message i; predicted[i] is the model's guess
actual    = ["spam", "spam", "not spam", "not spam", "spam",
             "not spam", "not spam", "spam", "not spam", "not spam"]
predicted = ["spam", "not spam", "not spam", "not spam", "spam",
             "spam", "not spam", "spam", "not spam", "not spam"]

TP = FN = FP = TN = 0

for a, p in zip(actual, predicted):
    if a == "spam" and p == "spam":
        TP += 1
    elif a == "spam" and p == "not spam":
        FN += 1
    elif a == "not spam" and p == "spam":
        FP += 1
    else:  # a == "not spam" and p == "not spam"
        TN += 1

print("TP:", TP, " FN:", FN, " FP:", FP, " TN:", TN)
print("Accuracy :", round((TP + TN) / len(actual), 2))
print("Precision:", round(TP / (TP + FP), 2))
print("Recall   :", round(TP / (TP + FN), 2))

Trace it by hand alongside the code, message by message: message 1 is spam/spam → TP. Message 2 is spam/not-spam → FN. Messages 3 and 4 are not-spam/not-spam → TN, TN. Message 5 is spam/spam → TP. Message 6 is not-spam/spam → FP. Message 7 is not-spam/not-spam → TN. Message 8 is spam/spam → TP. Messages 9 and 10 are not-spam/not-spam → TN, TN. Final tally: TP=3, FN=1, FP=1, TN=5. The program prints:

TP: 3  FN: 1  FP: 1  TN: 5
Accuracy : 0.8
Precision: 0.75
Recall   : 0.75

In practice, you would use a tested library instead of writing this loop yourself. The scikit-learn library's confusion_matrix function does the same counting:

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(actual, predicted, labels=["spam", "not spam"])
print(cm)

Because we passed labels=["spam", "not spam"], the function builds its rows and columns in that order — row 0 / column 0 is "actual spam, predicted spam." The output is:

[[3 1]
 [1 5]]

Read this exactly like our hand-built table: row 0 is the "actual spam" row, showing [TP=3, FN=1]; row 1 is the "actual not spam" row, showing [FP=1, TN=5]. It matches our manual trace exactly, which is a good way to sanity-check that you understand what the library is doing rather than treating it as a black box.

Beyond Two Classes: The Multi-Class Confusion Matrix

Not every classifier is a yes/no decision. After Class 10, CBSE students choose a stream — Science, Commerce, or Arts. Suppose a school builds a (fairly crude) model that tries to predict which stream a student will choose, based on their Class 10 marks pattern, and the school tests it on 90 students whose actual choices are already known: 30 chose Science, 30 chose Commerce, 30 chose Arts.

A multi-class confusion matrix is no longer 2×2 — it becomes 3×3, one row and one column per class, but the idea is identical: each row is an actual class, each column is a predicted class, and each student lands in exactly one cell.

• Of the 30 students who actually chose Science, the model predicted Science for 25, Commerce for 4, and Arts for 1.
• Of the 30 students who actually chose Commerce, the model predicted Science for 3, Commerce for 24, and Arts for 3.
• Of the 30 students who actually chose Arts, the model predicted Science for 0, Commerce for 2, and Arts for 28.

The cells where actual class equals predicted class (25, 24, 28) form the diagonal of the matrix — these are every student the model got right. Overall accuracy is still just correct predictions over total predictions:

Accuracy = (25 + 24 + 28) / 90 = 77 / 90 ≈ 0.856  →  85.6%

But look at what the diagonal alone would hide: the model confuses Commerce with Science and Arts more than it confuses Science or Arts with anything else (Commerce has 6 wrong predictions out of 30, versus Science's 5 and Arts's 2). A single accuracy figure of 85.6% cannot tell you that Commerce is the shakiest prediction — only the full matrix, row by row, can. This is the deeper reason confusion matrices scale so well as an evaluation tool: whether you have 2 classes or 20, the matrix always tells you not just how often the model is wrong, but exactly which classes it mixes up with which, information a single percentage can never carry.

Check Your Understanding

Question 1. A COVID-19 rapid test is evaluated on 500 people, of whom 40 truly have the infection. The test's results give TP = 34, FN = 6, FP = 25, TN = 435. Compute accuracy, precision, and recall, and state in one sentence which of the two error types (FP or FN) is more dangerous for a disease test, and why.

Answer: Accuracy = (34+435)/500 = 469/500 = 93.8%. Precision = 34/(34+25) = 34/59 ≈ 57.6%. Recall = 34/(34+6) = 34/40 = 85%. A false negative (FN) is the more dangerous error here — it tells an infected person they are healthy, so they may not isolate and could spread the infection, whereas a false positive (FP) only causes an unnecessary follow-up test.

Question 2. Two spam filters are compared on the same 1,000-message test set. Filter A has TP=180, FN=20, FP=5, TN=795. Filter B has TP=195, FN=5, FP=90, TN=710. Both were built for a hostel's shared Wi-Fi network where important exam-hall-ticket emails must never be missed. Which filter would you recommend, and which single metric best supports your choice?

Answer: Recall(A) = 180/200 = 90%; Recall(B) = 195/200 = 97.5%. Since the priority is never missing an important message (a false negative is the costly error here, exactly as it would be if an exam hall ticket got auto-deleted), Filter B's higher recall makes it the better choice for this use case — even though its precision (195/285 ≈ 68.4%) is worse than Filter A's (180/185 ≈ 97.3%), meaning B will also throw more genuine messages into the spam folder as a side effect. This trade-off is worth stating explicitly: no single filter maximizes both metrics at once.

Question 3. Without doing any calculation, explain why a confusion matrix with TN much larger than every other cell can still hide a bad model, using the accuracy-trap idea from this chapter.

Answer: If the negative class is much more common than the positive class (as with fraud, rare disease, or rare defect detection), a model can score a huge TN count and therefore high accuracy simply by rarely or never predicting the positive class — even with TP near zero, meaning it fails at detecting the very thing it was built to detect. Only checking recall (and precision) directly, not accuracy, catches this.

Summary

A confusion matrix is a 2×2 table (or larger, for more classes) that cross-tabulates a classifier's predictions against the true labels, splitting every result into True Positives, True Negatives, False Positives, and False Negatives. From these four counts you can derive every standard evaluation metric: accuracy (overall correctness), precision (trustworthiness of positive predictions), recall (fraction of real positives actually caught), specificity (fraction of real negatives correctly left alone), and F1 (a balance of precision and recall). The chapter's central lesson is that a single accuracy number can be dangerously misleading whenever the classes are imbalanced — a model can score 99%+ accuracy while catching zero of the cases that actually matter, which is why reporting the full confusion matrix, not just accuracy, is considered best practice for evaluating any real classifier, from spam filters and fraud detectors to medical diagnostic tools and multi-class stream-prediction systems.