Beyond Accuracy: Precision, Recall, F1, and AUC-ROC

The 98% Accurate Model That Never Catches a Thief

Suppose the National Payments Corporation asks you to build a model that flags fraudulent UPI transactions. You test it on 1,000 real transactions from last month, of which 20 were confirmed fraud and 980 were legitimate. Your model achieves 98% accuracy. Impressive — until you check what it is actually predicting, and discover it labels every single transaction "legitimate," always, no matter what. It never once looks at the data. It cannot catch a single thief, and it is still right 98% of the time, because fraud is rare and accuracy only asks "how often was the prediction correct overall?"

This is not a contrived edge case. It is the normal situation for almost every classifier that matters in practice: rare-disease screening, spam filtering, defect detection on a production line, cancelled-flight prediction. Whenever the two classes are imbalanced, accuracy stops measuring what you actually care about. You need numbers that separately answer two different questions: when the model raises an alarm, how often is it right? And of all the real frauds sitting in the data, how many did it actually find? Those two questions are precision and recall, and this chapter builds them from scratch, along with the F1 score that combines them and the ROC/AUC framework that describes a model's behavior across every possible decision threshold at once.

Four Numbers Accuracy Throws Away: The Confusion Matrix

Accuracy compresses a classifier's entire performance into one fraction: correct predictions over total predictions. To do better, you first have to stop compressing. For a binary classifier (fraud vs. not-fraud, positive vs. negative), every single prediction falls into exactly one of four buckets, formed by crossing what actually happened with what the model predicted:

• True Positive (TP) — actually fraud, model said fraud. A catch.
• False Negative (FN) — actually fraud, model said legit. A miss.
• False Positive (FP) — actually legit, model said fraud. A false alarm.
• True Negative (TN) — actually legit, model said legit. A correct pass-through.

Laid out as a 2×2 grid — actual class on the rows, predicted class on the columns — this is the confusion matrix, and it is the object every other metric in this chapter is computed from. Suppose your real classifier, tested on the same 1,000 transactions, produces this matrix instead of the lazy always-legit one:

Add TP + TN and divide by the total and you get accuracy: (14 + 950)/1000 = 96.4%. That single number tells you almost nothing about the thing you actually built this model for — catching fraud. To see that, you need to split the matrix into two separate ratios.

Precision: How Much Do You Trust an Alarm?

Look only at the column where the model predicted "fraud." It contains 14 true positives and 30 false positives — 44 alarms total. Precision asks: of everything the model flagged, what fraction was actually fraud?

Precision = TP / (TP + FP) = 14 / (14 + 30) = 14 / 44 = 0.318 = 31.8%

Precision is a statement about the model's alarms, not about the world. It is the conditional probability that a transaction is genuinely fraudulent, given that the model said "fraud": Precision = P(Actual = Fraud | Predicted = Fraud). At 31.8%, roughly two out of every three fraud alerts this model raises are false alarms on innocent transactions — each one potentially a blocked payment, a frozen account, an angry customer calling the bank. Low precision has a real cost even when recall is good.

Recall: How Many Thieves Actually Get Caught?

Now look only at the row where fraud actually happened. It contains 14 true positives and 6 false negatives — 20 real frauds total. Recall (also called sensitivity, or the True Positive Rate) asks: of everything that was actually fraud, what fraction did the model catch?

Recall = TP / (TP + FN) = 14 / (14 + 6) = 14 / 20 = 0.70 = 70%

Recall is a statement about reality, not about the model's alarms: Recall = P(Predicted = Fraud | Actual = Fraud). This model catches 70% of real fraud and lets 30% through completely undetected — six frauds a month sail past it with no alert at all. Notice that precision and recall are computed from different bases (44 alarms vs. 20 real frauds) and answer genuinely different questions. A model can be excellent by one measure and mediocre by the other, and accuracy would never have told you which.

The Threshold Dial: Why Precision and Recall Fight Each Other

Real classifiers do not output "fraud" or "legit" directly. They output a probability-like score between 0 and 1, and a separate decision rule — the threshold — converts that score into a label. "Predict fraud if score ≥ 0.5" is the default, but 0.5 is just a choice, and moving it trades precision against recall. Take ten transactions, scored by the same model and sorted from most to least suspicious:

Score  0.95  0.90  0.85  0.78  0.65  0.55  0.50  0.40  0.30  0.10
Label     F     F     L     F     L     F     L     L     L     L
(F = actually fraud, L = actually legitimate; 4 fraud, 6 legitimate)

Pick three different thresholds and recompute the confusion matrix at each one:

Raise the threshold and the model only flags transactions it is very sure about: it makes few false alarms, so precision is high, but it also misses fraud that scored just below the bar, so recall is low. Lower the threshold and it flags almost everything suspicious-looking: it catches nearly all the real fraud (recall climbs), but drags in far more false alarms (precision falls). One fact is worth stating precisely, because it is provable and it is tested: lowering the threshold can never decrease recall. Every transaction that was flagged at the old, higher threshold is still flagged at the new, lower one — TP can only stay the same or grow, so recall (TP over the fixed count of real frauds) can only stay the same or grow. Precision has no such guarantee; it can rise, fall, or zig-zag as the threshold moves, because both TP and FP are changing at once. There is no threshold that maximizes both precision and recall simultaneously for a model that isn't perfect — you are choosing a point on a trade-off curve, not eliminating it.

One Number to Rule Them: Deriving the F1 Score

Comparing two models by two separate numbers is awkward, so it is tempting to average precision and recall. The obvious average — the arithmetic mean — is misleading. Consider a model with Precision = 100% and Recall = 1%: it flags almost nothing, but everything it does flag is correct. Its arithmetic mean is (1.00 + 0.01)/2 = 0.505, a respectable-looking 50.5%, even though the model is close to useless — it misses 99 out of every 100 frauds. The arithmetic mean is fooled because it lets one large number hide one tiny one.

What you want instead is a mean that stays low whenever either input is low. That is exactly what the harmonic mean does, and the F1 score is the harmonic mean of precision and recall:

F1 = 2 / (1/P + 1/R)

To see why this punishes imbalance, simplify it algebraically. Put the two reciprocals over a common denominator:

1/P + 1/R = R/(PR) + P/(PR) = (P + R)/(PR)

Now invert and multiply by 2:

F1 = 2 / [(P + R)/(PR)] = 2PR / (P + R)

That is the working formula: twice the product, divided by the sum. Test it on the extreme case above, P = 1.00, R = 0.01: F1 = 2(1.00)(0.01)/(1.01) = 0.02/1.01 ≈ 1.98%. The harmonic mean correctly reports a nearly-useless model, where the arithmetic mean reported a mediocre-but-acceptable 50.5%. This is the core reason F1, not the plain average, is the standard single-number summary of a classifier: it can only be high when both precision and recall are reasonably high, and it collapses toward the smaller of the two whenever they are far apart.

Apply it to the real fraud model: P = 14/44 = 7/22, R = 14/20 = 7/10.

PR = (7/22)(7/10) = 49/220
P + R = 7/22 + 7/10 = (35 + 77)/110 = 112/110 = 56/55
F1 = 2 · (49/220) / (56/55) = (98/220) · (55/56) = 98/224 = 0.4375

F1 = 43.75%. Compare that to the misleadingly comfortable arithmetic mean, (0.318 + 0.70)/2 = 50.9% — F1 correctly reflects that this model's biggest weakness, its poor precision, is dragging its overall usefulness down, not just averaging it away. You can check the whole chain with three lines of code:

TP, FP, FN = 14, 30, 6

precision = TP / (TP + FP)
recall    = TP / (TP + FN)
f1        = 2 * precision * recall / (precision + recall)

print(f"Precision: {precision:.3f}")
print(f"Recall:    {recall:.3f}")
print(f"F1 score:  {f1:.3f}")

Tracing it: precision = 14/44 = 0.31818…, printed as 0.318. recall = 14/20 = 0.7, printed as 0.700. f1 = 2 × 0.31818… × 0.7 ÷ (0.31818… + 0.7) = 0.44545…/1.01818… = 0.4375 exactly, printed as 0.438. Output:

Precision: 0.318
Recall:    0.700
F1 score:  0.438

F-beta: Telling the Model Which Mistake You Fear More

Plain F1 treats a missed fraud and a false alarm as equally bad. They rarely are. In a rural TB-screening camp, a missed case (false negative) can mean an undiagnosed, spreading infection — far worse than one extra healthy patient sent for a confirmatory test (false positive). In an email spam filter, a false positive that buries a JEE admit-card email in spam can be worse than one spam message slipping through. The F-beta score generalizes F1 with a weight β that controls which error matters more:

F_beta = (1 + beta^2) · P · R / (beta^2 · P + R)

Set β = 1 and this collapses back to the F1 formula above — check it: (1+1)PR/(P+R) = 2PR/(P+R). Setting β > 1 (commonly β = 2) weights recall more heavily, appropriate for the TB-screening case where missing a real case is the worse mistake. Setting β < 1 (commonly β = 0.5) weights precision more heavily, appropriate when false alarms are the costlier mistake. On the fraud model (P = 0.318, R = 0.70):

F2   (recall matters more)   = 5PR/(4P+R)    = 0.565 = 56.5%
F0.5 (precision matters more) = 1.25PR/(0.25P+R) = 0.357 = 35.7%

F2 pulls the score up toward recall (70%) because it rewards the model for catching fraud even at the cost of false alarms; F0.5 pulls it down toward precision (31.8%) because it punishes the model harder for those same false alarms. Neither answer is "more correct" than the other — β encodes a real business or medical decision about which error is more expensive, and that decision has to be made by a human, not read off a formula.

Building the ROC Curve, Point by Point

F-beta still forces you to commit to one threshold and one β before you can compute anything. The ROC curve (Receiver Operating Characteristic) instead shows how a model behaves across every possible threshold at once, by plotting two rates against each other:

TPR (True Positive Rate) = TP/(TP+FN)   -- this IS Recall
FPR (False Positive Rate) = FP/(FP+TN)  -- fraction of legit transactions wrongly flagged

Build it directly from the ten sorted scores used earlier. Start at the strictest possible threshold (nothing flagged, point (0,0)) and walk down the sorted list one transaction at a time, lowering the threshold past each score in turn. Each time you pass a real fraud, TP rises by one, so TPR steps up by 1/4 (there are 4 real frauds). Each time you pass a legitimate transaction, FP rises by one, so FPR steps right by 1/6 (there are 6 legitimate transactions):

Order: F   F   L    F     L     F    L    L    L    L
Step:  up  up  rt   up    rt    up   rt   rt   rt   rt
Point: (0,.25) (0,.5) (.167,.5) (.167,.75) (.333,.75) (.333,1) (.5,1) (.667,1) (.833,1) (1,1)

Plotted, this traces a staircase from the bottom-left corner (0,0) to the top-right corner (1,1):

Every point on this staircase is one specific threshold's (FPR, TPR) pair — the three thresholds worked out earlier all live on this same curve. A model with zero discriminating power (scores unrelated to the true label) produces a curve that hugs the dashed diagonal, because at every threshold, the fraction of frauds caught roughly equals the fraction of legit transactions wrongly flagged. A model with real signal bows up and to the left, toward the corner where TPR = 1 and FPR = 0 — catch everything, flag nothing extra. The whole curve, not any single point on it, is what an ROC comparison is about.

AUC: The Area That Means Something

Collapsing the whole curve into one number gives the Area Under the Curve (AUC). Since this ROC curve is a staircase, its area is a sum of rectangles: each rightward (FPR-increasing) step contributes width × height, where height is the TPR level at that step (vertical steps add zero width, hence zero area):

Step to (.167,.5):  width 1/6, height .5  -> area .0833
Step to (.333,.75): width 1/6, height .75 -> area .1250
Step to (.5,1):      width 1/6, height 1  -> area .1667
Step to (.667,1):    width 1/6, height 1  -> area .1667
Step to (.833,1):    width 1/6, height 1  -> area .1667
Step to (1,1):        width 1/6, height 1  -> area .1667
                                    Total AUC = 0.875

AUC = 0.875 has a second, more useful meaning worth deriving independently as a check: AUC equals the probability that a randomly chosen positive example scores higher than a randomly chosen negative example. Verify this directly by pair-counting. There are 4 positive scores {0.95, 0.90, 0.78, 0.55} and 6 negative scores {0.85, 0.65, 0.50, 0.40, 0.30, 0.10}, giving 4 × 6 = 24 possible pairs. Count how many have the positive score higher: 0.95 beats all 6 negatives; 0.90 beats all 6; 0.78 beats 5 (loses only to 0.85); 0.55 beats 4 (loses to 0.85 and 0.65). Total = 6+6+5+4 = 21, and 21/24 = 0.875 — exactly matching the trapezoid calculation. This equivalence is not a coincidence; it is the Mann-Whitney U statistic in disguise, and it is why AUC = 0.5 means the model ranks positives and negatives no better than a coin flip, AUC = 1.0 means every positive outranks every negative (perfect separation), and AUC below 0.5 means the model's ranking is worse than random — inverting its predictions would help.

Common Misconception: "A Better Model Has Higher Precision AND Higher Recall"

Students often assume that improving a model means both precision and recall should go up together, and that if one model has higher recall than another, it must be "trying harder" and therefore worse on precision by nature. Neither is quite right, and the distinction matters. Moving the threshold on one fixed, trained model slides you along that model's single ROC curve — you trade recall for precision or vice versa, but you have not changed the model's underlying ability to separate fraud from non-fraud, and the AUC stays exactly the same. Training a genuinely better model — more data, better features, a stronger algorithm — shifts the entire curve up and to the left, increasing AUC, so that at any FPR you choose, TPR is higher than before, and a threshold now exists that gives you better precision and better recall simultaneously than the old model could offer at any threshold. Threshold-tuning optimizes where you stand on a fixed curve; model improvement redraws the curve itself. Confusing the two leads to a common mistake: reporting one precision/recall pair from a threshold you happened to pick, and treating it as a complete performance claim, when a competitor's model with a lower AUC could easily beat it at that one specific threshold while being worse everywhere else.

Where This Shows Up: CBSE and Competitive Exams

The confusion matrix, precision, recall, and F1 are explicit, named topics in the CBSE Artificial Intelligence curriculum's model-evaluation unit, and numeric confusion-matrix problems (compute precision/recall/F1 from given TP/FP/FN/TN, or build the matrix from a small labelled dataset) are standard board-exam and practical-file question types. Framing precision as P(Actual = Fraud | Predicted = Fraud) and recall as P(Predicted = Fraud | Actual = Fraud) is also a direct rehearsal for conditional-probability and Bayes'-theorem questions that appear across JEE Main, BITSAT, and KVPY — the notation and reasoning transfer exactly. GATE's Data Science and AI paper tests confusion-matrix arithmetic, F1/F-beta computation, and ROC/AUC interpretation directly at a slightly more formal level than boards; the harmonic-mean derivation shown here is exactly the algebra examiners expect you to be able to reproduce, not just quote.

Test Your Understanding

Summary

Accuracy answers one blunt question — how often was the model right overall — and hides how it fails on the class you actually care about, especially when that class is rare. The confusion matrix (TP, FP, FN, TN) recovers the detail accuracy discards. Precision = TP/(TP+FP) measures how trustworthy a positive prediction is; Recall = TP/(TP+FN) measures how much of the real positive class the model actually finds; moving the classification threshold trades one against the other along one fixed model's performance, and lowering the threshold can only ever raise or hold recall, never lower it. F1 = 2PR/(P+R), the harmonic mean of the two, is the honest single-number summary because it collapses toward whichever of P or R is smaller, unlike a plain average; F-beta = (1+β²)PR/(β²P+R) generalizes F1 to let you state, numerically, which kind of mistake is worse. The ROC curve plots TPR against FPR across every threshold at once, built by stepping up for each true positive and right for each false positive as the threshold is lowered through sorted scores; AUC, the area under that curve, equals the probability that a random positive example outranks a random negative one, runs from 0.5 (no better than a coin flip) to 1.0 (perfect separation), and — unlike a single precision/recall pair — measures the model's underlying ranking ability independent of any one threshold choice.

Think About It

Think about this: How would you explain beyond accuracy: precision, recall, f1, and auc-roc to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.