Model Evaluation: Beyond Accuracy — Precision, Recall, F1, and ROC

Suppose you build a machine learning model to catch fraudulent UPI transactions. Your bank processes 100,000 transactions today. Historically, about 100 of every 100,000 transactions are fraud — the rest, 99,900, are genuine. You train a model, run it on today's data, and it reports 99.9% accuracy. Your manager is impressed. Should she be?

Here is the uncomfortable fact: a model that does nothing — that simply predicts "not fraud" for every single transaction, without looking at the data at all — also scores 99.9% accuracy on this exact dataset. It gets all 99,900 genuine transactions right and all 100 fraud cases wrong, and 99,900 out of 100,000 is still 99.9%. That "do-nothing" model is completely useless — it has never caught a single fraudulent transaction in its life — yet accuracy cannot tell it apart from a genuinely good model. This is called the accuracy paradox, and it is the single biggest reason accuracy is the wrong metric whenever one class is rare. Fraud detection, disease screening, spam filtering, defect detection on a factory line — in every one of these, the class you actually care about is the rare one, and accuracy quietly rewards ignoring it. This chapter builds the tools that replace accuracy: precision, recall, F1, and the ROC curve. Each one answers a specific, honest question that accuracy refuses to ask.

The Confusion Matrix: Four Numbers Instead of One

Before any of these metrics can be defined, every prediction a classifier makes has to be sorted into exactly one of four buckets, by comparing what the model predicted against what actually happened:

• True Positive (TP) — model predicted fraud, and it really was fraud.
• False Positive (FP) — model predicted fraud, but it was actually a genuine transaction (a false alarm).
• False Negative (FN) — model predicted "not fraud," but it actually was fraud (a miss).
• True Negative (TN) — model predicted "not fraud," and it really was genuine.

Arrange these four counts in a 2×2 grid — rows for what actually happened, columns for what the model predicted — and you get the confusion matrix. Every metric in this chapter is built purely from these four numbers.

For the do-nothing model on our 100,000-transaction day: TP = 0 (it never predicted fraud, so it can never be right about fraud), FP = 0, FN = 100 (every real fraud case was missed), TN = 99,900. Accuracy = (TP + TN) / total = 99,900 / 100,000 = 99.9%, exactly as before — but now you can also see TP = 0, which is the number that actually mattered and accuracy hid.

Precision: When You Cry Fraud, How Often Are You Right?

Now consider two real models built for this problem.

Model A is conservative: it flags a transaction as fraud only when it is very confident. Today it flagged 40 transactions. Investigating them, the bank finds 35 were genuinely fraudulent and 5 were false alarms. Since only 100 fraud cases existed, it missed 65 of them.

So for Model A: TP = 35, FP = 5, FN = 65, TN = 99,895.

Precision answers: "Of everything the model called fraud, what fraction actually was fraud?"

Precision = TP / (TP + FP)

For Model A: Precision = 35 / (35 + 5) = 35 / 40 = 0.875, or 87.5%. When this model raises an alarm, it is right seven times out of eight. That matters directly in rupees and customer trust — every false alarm means blocking a genuine transaction and a frustrated customer calling the bank.

Recall: Of All the Fraud That Happened, How Much Did You Catch?

Model B takes the opposite approach: it flags anything even slightly suspicious. Today it flagged 300 transactions. Of these, 90 were real fraud and 210 were false alarms. Since 90 of the 100 real fraud cases were caught, 10 were missed.

So for Model B: TP = 90, FP = 210, FN = 10, TN = 99,690.

Recall (also called sensitivity) answers a different question: "Of all the fraud that actually happened, what fraction did the model catch?"

Recall = TP / (TP + FN)

For Model B: Recall = 90 / (90 + 10) = 90 / 100 = 0.90, or 90%. Compare Model A's recall: 35 / (35 + 65) = 35/100 = 0.35, or just 35%.

Now compute Model B's precision: 90 / (90 + 210) = 90/300 = 0.30, or 30%. Only 3 out of every 10 alarms Model B raises are real fraud — 7 out of 10 are innocent customers getting blocked.

This is the fundamental tension in classification: Model A has high precision but low recall. Model B has high recall but low precision. Neither number alone tells you which model is better — it depends on what a missed fraud costs the bank versus what an angry, wrongly-blocked customer costs. A single number that balances both is useful, which is exactly what F1 is for.

The Misconception: F1 Is Not the Average of Precision and Recall

The natural instinct is to average precision and recall — add them and divide by 2. This instinct is wrong, and seeing why is one of the most important ideas in this chapter.

Consider an extreme Model C that is so cautious it flags only a single transaction all day — and gets it right. TP = 1, FP = 0, FN = 99 (the other 99 fraud cases go completely uncaught), TN = 99,900. Precision = 1/(1+0) = 1.0, a perfect 100%. Recall = 1/(1+99) = 0.01, a dismal 1%. This model is nearly worthless — it catches almost no fraud — yet the simple average of its precision and recall is (1.0 + 0.01)/2 = 0.505, which looks like a decent 50%. The arithmetic mean is fooled by the one huge number and completely hides the catastrophic recall.

F1 score is instead the harmonic mean of precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

For Model C: F1 = 2 × (1.0 × 0.01) / (1.0 + 0.01) = 0.02 / 1.01 ≈ 0.0198, or about 2%. This is the honest picture — a model this lopsided deserves a score near zero, not near 50%.

Why does the harmonic mean do this? Algebraically, the harmonic mean of two positive numbers is always less than or equal to their arithmetic mean, and it leans hard toward whichever number is smaller. If either precision or recall collapses toward 0, the product term (Precision × Recall) collapses toward 0 much faster than the sum term shrinks, so F1 collapses too — exactly the punishing behaviour you want when one of the two failure modes (false alarms or missed fraud) is being ignored entirely. The arithmetic mean has no such penalty; it treats "1.0 and 0.01" the same as "0.505 and 0.505," which is precisely the confusion F1 exists to prevent.

Now the balanced comparison you actually want, for Models A and B:

Model A: F1 = 2 × (0.875 × 0.35) / (0.875 + 0.35) = 0.6125 / 1.225 = 0.50.
Model B: F1 = 2 × (0.30 × 0.90) / (0.30 + 0.90) = 0.54 / 1.2 = 0.45.

By F1, Model A edges out Model B — but notice this doesn't automatically make Model A the "correct" choice for the bank. F1 silently assumes a false alarm and a missed fraud cost the business equally. If missing a ₹2,00,000 fraudulent transfer is far more costly than one customer's card being temporarily blocked, the bank may deliberately prefer Model B's higher recall even though its F1 is lower. F1 is a good default single-number summary — it is not a substitute for deciding, explicitly, which error type is more expensive.

Choosing a Threshold: Precision and Recall Are a Dial, Not Fixed Numbers

Models A and B were not built with different algorithms — the same underlying model can behave like either one. A classifier does not output "fraud" or "not fraud" directly; it outputs a probability that a transaction is fraudulent, and a chosen threshold converts that probability into a decision. "Flag as fraud if probability ≥ 0.5" is one choice; "flag if probability ≥ 0.2" is another, more aggressive choice that will catch more fraud (higher recall) at the cost of more false alarms (lower precision). Raising the threshold moves you toward Model A's behaviour; lowering it moves you toward Model B's. Precision and recall are not properties of a model — they are properties of a model at a specific threshold.

This raises a natural question: instead of picking one threshold and reporting one precision-recall pair, can you see the model's behaviour across every possible threshold at once? That is exactly what the ROC curve does.

The ROC Curve: Every Threshold, Plotted at Once

Take ten transactions the model has scored with a fraud probability, sorted from most to least suspicious. Four are genuinely fraudulent (marked 1), six are genuine (marked 0):

Score:  0.95  0.90  0.85  0.80  0.65  0.55  0.40  0.30  0.20  0.10
Actual:   1     1     0     1     0     1     0     0     0     0

The ROC curve plots, for every possible threshold, two rates:

TPR (True Positive Rate) = TP / (TP + FN)   ← this is exactly Recall
FPR (False Positive Rate) = FP / (FP + TN)  ← fraction of genuine transactions wrongly flagged

Start with the threshold above every score (nothing flagged: TPR = 0, FPR = 0), then lower it past each score in turn, recomputing TP and FP as more transactions get flagged:

Threshold just below   TP  FP    TPR    FPR
0.95                     1   0   0.25   0.000
0.90                     2   0   0.50   0.000
0.85                     2   1   0.50   0.167
0.80                     3   1   0.75   0.167
0.65                     3   2   0.75   0.333
0.55                     4   2   1.00   0.333
0.40                     4   3   1.00   0.500
0.30                     4   4   1.00   0.667
0.20                     4   5   1.00   0.833
0.10                     4   6   1.00   1.000

There are 4 actual fraud cases in total (P = 4) and 6 genuine ones (N = 6), which is why TPR maxes out once all 4 frauds are caught and FPR maxes out once all 6 genuine transactions get swept up too.

A model that guesses randomly produces the diagonal dashed line — at any threshold it flags the same fraction of frauds and genuine transactions alike, so TPR always equals FPR. Our model's staircase bulges well above that diagonal, climbing to high TPR while FPR is still low, which is exactly what a useful classifier's ROC curve should do: catch most of the real positives before it starts drowning in false alarms.

AUC: Collapsing the Whole Curve into One Number

AUC (Area Under the ROC Curve) compresses the entire curve into a single score between 0 and 1. There's a clean way to compute it by hand that also explains what it means: pick one fraud case and one genuine transaction at random, and ask — did the model give the fraud case a higher score? AUC is exactly the probability that it did, estimated by checking every possible pair.

There are 4 fraud cases and 6 genuine ones, so 4 × 6 = 24 possible pairs. Check each fraud score against all 6 genuine scores (0.85, 0.65, 0.40, 0.30, 0.20, 0.10):

0.95 beats all 6 genuine scores          → 6/6 correct
0.90 beats all 6 genuine scores          → 6/6 correct
0.80 beats 5 of 6 (loses only to 0.85)   → 5/6 correct
0.55 beats 4 of 6 (loses to 0.85, 0.65)  → 4/6 correct

Total correct pairs = 6 + 6 + 5 + 4 = 21 out of 24
AUC = 21 / 24 = 0.875

You can cross-check this by measuring the actual area under the staircase in the diagram above — summing each horizontal step's width multiplied by its height gives the same 0.875. An AUC of 0.5 is exactly what random guessing produces (the diagonal); an AUC of 1.0 means the model ranks every single fraud case above every single genuine transaction with zero mistakes. Our 0.875 says the model is quite good at ranking, though not flawless: one fraud case (the 0.55 one) still scored lower than two genuine transactions.

The genuine advantage of AUC over a single precision/recall/F1 number is that it does not require you to have already picked a threshold — it measures how well the model separates the two classes across every threshold simultaneously, which is exactly what you want when comparing two models before deciding where to draw the line.

Here is the same example verified in code — trace through it and confirm the numbers by hand before you trust the output:

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

y_true  = [1, 1, 0, 1, 0, 1, 0, 0, 0, 0]
y_score = [0.95, 0.90, 0.85, 0.80, 0.65, 0.55, 0.40, 0.30, 0.20, 0.10]
y_pred  = [1 if s >= 0.5 else 0 for s in y_score]

print(confusion_matrix(y_true, y_pred))
print("Precision:", precision_score(y_true, y_pred))
print("Recall:", recall_score(y_true, y_pred))
print("F1 score:", f1_score(y_true, y_pred))

# Output:
# [[4 2]
#  [0 4]]
# Precision: 0.6666666666666666
# Recall: 1.0
# F1 score: 0.8

Trace it: at threshold 0.5, scores 0.95 down to 0.55 (six transactions) get predicted 1; the rest get predicted 0. Comparing to y_true, transactions 3 and 5 (both actually 0) are wrongly predicted 1 — that's FP = 2. All four actual 1s (transactions 1, 2, 4, 6) are correctly predicted 1 — TP = 4, FN = 0. The remaining four actual 0s are correctly predicted 0 — TN = 4. sklearn's confusion_matrix orders rows/columns as [0, 1], so it prints as [[TN, FP], [FN, TP]] = [[4, 2], [0, 4]], and Precision = 4/6 = 0.667, Recall = 4/4 = 1.0, F1 = 2(0.667)(1.0)/1.667 = 0.8 — matching the threshold-0.55 row of the table above exactly, since "≥ 0.5" and "≥ 0.55" flag the same six transactions here.

The Exam Connection: Precision Is a Conditional Probability

Precision has a name in probability theory: it is P(Actual Positive | Predicted Positive) — given that the model predicted fraud, what's the probability it really is fraud? Recall is the reverse conditional, P(Predicted Positive | Actual Positive). These are not the same quantity, and confusing the two is a classic error — it is exactly the mistake of confusing P(A|B) with P(B|A), which shows up constantly in probability problems.

In fact, you can derive precision directly from recall using Bayes' theorem. If Recall = P(Pred+|Actual+), and you also know the FPR = P(Pred+|Actual−), and the prevalence P(Actual+) — the fraction of all transactions that are truly fraud — then:

P(Actual+ | Pred+) = P(Pred+ | Actual+) × P(Actual+)
                      ────────────────────────────────
                      P(Pred+ | Actual+)×P(Actual+) + P(Pred+ | Actual−)×P(Actual−)

This is precision, rebuilt entirely from recall, FPR, and prevalence — the denominator is just P(Pred+) expanded using the law of total probability. When you reach the chapter on Conditional Probability and Bayes' Theorem in Class 12, you will recognize this exact structure immediately; the intuition you've already built here — that a rare condition (low prevalence, like 100 fraud cases in 100,000) makes even a fairly accurate detector produce mostly false alarms — is the same reasoning tested in JEE Main and Advanced probability questions, in CUET mathematics, and in CBSE board-level conditional probability problems once that chapter enters your syllabus. It is also the exact reasoning behind why a medical test with "99% accuracy" can still be wrong more often than right when screening for a rare disease — precision falls as prevalence falls, even while recall and FPR stay fixed.

Test Yourself: Compute It

1. A spam filter checks 2,000 emails. 40 are genuinely spam. The filter flags 55 emails as spam, and 32 of those really are spam. Compute precision, recall, and F1. (Answer: Precision = 32/55 ≈ 0.582; Recall = 32/40 = 0.80; F1 = 2(0.582)(0.80)/(0.582+0.80) ≈ 0.674.)

2. Explain, using the harmonic-mean formula, why a model with Precision = 0.9 and Recall = 0.1 gets an F1 far closer to 0.1 than to the arithmetic-mean value of 0.5. (Answer: F1 = 2(0.9)(0.1)/(0.9+0.1) = 0.18/1.0 = 0.18 — the harmonic mean is dragged toward the smaller of the two values because the product term shrinks proportionally to the smallest factor.)

3. A model scores five transactions — three genuine (scores 0.7, 0.3, 0.1) and two fraudulent (scores 0.9, 0.2) — for fraud. Using the pairwise method, compute the AUC by hand. (Answer: pairs = 3 × 2 = 6. Check 0.9 against 0.7, 0.3, 0.1: beats all 3. Check 0.2 against 0.7, 0.3, 0.1: beats only 0.1, so 1 correct. Total correct = 4/6 = 0.667.)

Summary

• Accuracy = (TP+TN)/Total is misleading whenever classes are imbalanced — a model that predicts the majority class every time can score high accuracy while catching zero of what you actually care about (the accuracy paradox).
• The confusion matrix (TP, FP, FN, TN) is the foundation every other metric is built from.
• Precision = TP/(TP+FP) — of everything flagged positive, how much really was positive. It punishes false alarms.
• Recall = TP/(TP+FN) — of everything that was truly positive, how much got caught. It punishes misses.
• F1 = 2 × Precision × Recall / (Precision + Recall) is the harmonic mean, not the arithmetic mean — it collapses toward zero if either precision or recall collapses, unlike a simple average which can be fooled by one large number.
• Precision and recall depend on a chosen classification threshold; moving that threshold trades one against the other.
• The ROC curve plots TPR (= Recall) against FPR across every possible threshold at once; AUC is the probability that a randomly chosen positive example is ranked above a randomly chosen negative one, computable by hand as (correctly ordered pairs) / (total pairs).
• Precision is the conditional probability P(Actual Positive | Predicted Positive) — the same Bayes'-theorem structure you'll formally meet in Class 12 conditional probability, and the same reasoning tested in JEE Main/Advanced and CUET mathematics.

Think About It

Think about this: How would you explain model evaluation: beyond accuracy — precision, recall, f1, and roc to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where model evaluation: beyond accuracy — precision, recall, f1, and roc is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting model evaluation: beyond accuracy — precision, recall, f1, and roc to at least 3 other topics you have studied.