Naive Bayes: Probabilistic Classification

Every time a message lands in your phone's spam folder, or a UPI app flags a transaction as suspicious before you've even noticed anything wrong, some piece of software has done something you can do by hand with a pencil and about ten multiplications. It looked at a handful of clues, asked "given these clues, which outcome is more probable?", and answered in microseconds. That question — turning scattered evidence into a probability-backed decision — is exactly what this chapter teaches you to compute yourself, starting from nothing but the definition of conditional probability.

The algorithm is called Naive Bayes. The name is almost a warning label: it is built on an assumption that is, strictly speaking, false in almost every real dataset — and yet it remains one of the fastest, most interpretable, and most surprisingly effective classifiers in machine learning. By the end of this chapter you will have derived the theorem it rests on from first principles, built two working classifiers by hand (one for continuous measurements, one for text), traced a working Python implementation line by line, and understood precisely why an assumption everyone agrees is wrong still produces a classifier that is very often right.

The Problem Naive Bayes Solves

Classification, stated formally, is this: you are given a set of possible classes C = {c1, c2, ..., ck} and a feature vector X = (x1, x2, ..., xn) describing one example. You must output the class the example most likely belongs to. A spam filter has two classes (Spam, Ham) and features drawn from the words in a message. A fruit-sorting machine might have two classes (Ripe, Unripe) and one feature — weight. A medical triage system might have many classes and features drawn from symptoms and test results.

What makes this a probabilistic classifier, rather than a rule-based one, is that Naive Bayes does not just output a class — it outputs a number for every class: the probability that the class is correct, given the evidence in front of it. It then picks whichever class has the highest probability. Everything in this chapter is really one question, asked over and over in different costumes: how do you compute P(class | evidence) when all your training data only lets you measure P(evidence | class) directly?

Building Intuition: Belief Before and After Evidence

Picture a mango vendor at a wholesale market. Before touching any single mango, she already has a rough belief: on a given morning, most of her stock — say 60% — tends to be ripe, and 40% unripe, just from how the batch was picked and transported. That 60/40 split is her prior belief, formed before she looks at any individual fruit.

Now she picks up one mango and presses it gently. It gives slightly — soft, not hard. She knows from years of experience that ripe mangoes give like this far more often than unripe ones do. That single piece of evidence doesn't erase her prior belief; it updates it. Her confidence that this particular mango is ripe goes up, because the evidence (softness) is more typical of the Ripe class than the Unripe class.

This two-step move — start with a prior, fold in evidence, land on an updated ("posterior") belief — is the entire logical skeleton of Naive Bayes. Everything that follows in this chapter is about making that intuitive move mathematically precise and computable, so that instead of "years of experience," the classifier uses counts and measurements from training data.

Deriving Bayes' Theorem From First Principles

We need one definition to start: for two events A and B with P(B) > 0, the conditional probability of A given B is defined as

P(A|B) = P(A and B) / P(B)

This is just saying "restrict your attention to the world where B happened, and ask what fraction of that restricted world also has A happening." By the identical logic, swapping the roles of A and B,

P(B|A) = P(A and B) / P(A)

Both expressions describe the same quantity, P(A and B), just measured two different ways. Rearranging each to isolate that shared quantity:

P(A and B) = P(A|B) · P(B)
P(A and B) = P(B|A) · P(A)

Since both right-hand sides equal the same thing, we can set them equal to each other:

P(A|B) · P(B) = P(B|A) · P(A)

Dividing both sides by P(B) gives Bayes' theorem:

P(A|B) = P(B|A) · P(A) / P(B)

That's the whole derivation — two applications of one definition and one rearrangement. No step required anything beyond what "conditional probability" already means. (This exact theorem reappears formally in the CBSE Class 12 Probability chapter; you're getting a complete, self-contained derivation of it now, so nothing here depends on having seen it before.)

Let's make it concrete with numbers before we ever mention classifiers. A school runs a coding test across two sections. Section 10-A has 40 students, of whom 18 passed. Section 10-B has 35 students, of whom 28 passed. Suppose someone picks a section by a fair coin flip (so P(10-A) = P(10-B) = 0.5) and then names a random student from that section, and that student turns out to have passed. What's the probability the student came from 10-A?

We have P(Pass|10-A) = 18/40 = 0.45 and P(Pass|10-B) = 28/35 = 0.8. We need P(Pass), the total probability of drawing a pass regardless of section. This comes from the law of total probability: weight each section's pass rate by the chance of landing in that section.

P(Pass) = P(Pass|10-A)·P(10-A) + P(Pass|10-B)·P(10-B)
        = 0.45 × 0.5 + 0.8 × 0.5
        = 0.225 + 0.4 = 0.625

Now apply Bayes' theorem:

P(10-A|Pass) = P(Pass|10-A) · P(10-A) / P(Pass)
             = (0.45 × 0.5) / 0.625
             = 0.225 / 0.625 = 0.36

So even though the coin gave each section an equal 50% chance of being picked, learning that the student passed shifts our belief: there's only a 36% chance they're from 10-A and a 64% chance they're from 10-B, because 10-B's much higher pass rate makes "passed" far more typical evidence for 10-B. Notice the mechanics: prior (0.5) times likelihood (0.45) gives an unnormalized score, and dividing by the total probability of the evidence turns that score into a genuine probability. That normalizing step is exactly what you'll do for every classifier in this chapter.

From One Theorem to a Classifier

Relabel the pieces. Instead of "section," write C for class. Instead of "passed the test," write X for the observed features. Bayes' theorem becomes:

P(C|X) = P(X|C) · P(C) / P(X)

P(C) is the prior — how common each class is before you look at any features (estimated as the fraction of training examples in that class). P(X|C) is the likelihood — how probable this exact evidence is if the example really belongs to class C (estimated from how that evidence looked across training examples of that class). P(X) is the probability of seeing this evidence at all, regardless of class.

Here's the shortcut that makes classification cheap: P(X) does not depend on which class you're evaluating — it's the same number no matter which C you plug in. Since we only care about which class has the highest posterior, not the exact value, we can skip computing it and just compare unnormalized scores:

P(C|X) ∝ P(X|C) · P(C)

The classifier's decision rule is then to pick whichever class maximizes this product — written argmax, "the class that makes this expression largest":

predicted class = argmax over C of [ P(X|C) · P(C) ]

If you do want actual probabilities (not just a winner), you compute the score for every class and divide each by the sum of all the scores — precisely the normalizing step from the section-passing example, just applied across however many classes you have.

The "Naive" Assumption: Taming a Combinatorial Explosion

There is a serious problem hiding inside P(X|C) once X stops being a single number and becomes a vector of many features (x1, x2, ..., xn). By the chain rule of probability, the exact joint likelihood expands as

P(x1,...,xn | C) = P(x1|C) · P(x2|x1,C) · P(x3|x1,x2,C) · ... · P(xn|x1,...,x(n-1),C)

Every term after the first has to account for every combination of the features before it. If you had just 20 binary (yes/no) features, specifying this joint distribution exactly would require estimating close to 2^20 - 1 ≈ 1,048,575 independent probabilities per class from your training data — and most real datasets don't have a million examples per class to estimate that many numbers reliably.

The "naive" step is to assume every feature is conditionally independent of every other feature, given the class:

P(xi | x1,...,x(i-1), C) = P(xi | C) for every feature i

Under this assumption the entire product collapses to

P(x1,...,xn | C) = P(x1|C) · P(x2|C) · ... · P(xn|C) = ∏ P(xi|C)

Instead of roughly a million parameters, you now need just 20 — one per feature, per class. That collapse from exponential to linear is the entire reason Naive Bayes is fast to train, fast to run, and works even with modest amounts of data. It is also, obviously, not true: in a spam email, the words "free" and "prize" are far more likely to co-occur than independence would predict. The assumption is a simplification made in exchange for tractability, and the question of what that trade costs you is exactly what the misconception section further down addresses.

Worked Example 1 — Gaussian Naive Bayes: Sorting Mangoes by Weight

Return to the vendor's mangoes, now with real numbers. Her training records, built from many past mangoes she weighed and later confirmed as ripe or unripe, give:

• Prior: P(Ripe) = 0.6, P(Unripe) = 0.4 (60% of past mangoes were ripe)
• Ripe class weight: mean μ = 200 g, standard deviation σ = 8 g
• Unripe class weight: mean μ = 150 g, standard deviation σ = 10 g

Weight is a continuous number, not a count, so we can't use frequency counting the way we will for text in Example 2. Instead, Gaussian Naive Bayes assumes each class's feature values follow a normal (bell-curve) distribution, and uses the Gaussian probability density function as the likelihood:

P(x|C) = 1 / sqrt(2πσ²) × exp( -(x-μ)² / (2σ²) )

A new mango weighs 175 g. We classify it by computing the likelihood under each class.

Ripe: σ² = 64. The coefficient is 1/sqrt(2π×64) = 1/sqrt(402.12) = 1/20.053 ≈ 0.04987. The exponent: x - μ = 175 - 200 = -25, so (x-μ)² = 625, and 625 / (2×64) = 625/128 = 4.8828. So exp(-4.8828) ≈ 0.007575. Multiplying: P(175|Ripe) ≈ 0.04987 × 0.007575 ≈ 0.0003777.

Unripe: σ² = 100. Coefficient: 1/sqrt(2π×100) = 1/sqrt(628.32) = 1/25.066 ≈ 0.03989. Exponent: x - μ = 175 - 150 = 25, (x-μ)² = 625, and 625/(2×100) = 625/200 = 3.125, so exp(-3.125) ≈ 0.04394. Multiplying: P(175|Unripe) ≈ 0.03989 × 0.04394 ≈ 0.0017532.

Notice something worth pausing on: 175 g is exactly 25 g from both means — equidistant. If the two classes had identical spread, the likelihoods would tie. They don't tie, because Unripe has a larger standard deviation (10 g vs. 8 g): its bell curve is wider and flatter, so it retains more density farther from its peak. That's why P(175|Unripe) comes out roughly 4.6 times larger than P(175|Ripe), even at equal distance. The diagram below makes this visible.

Now bring in the priors and normalize. The unnormalized posterior scores are P(Ripe)×P(175|Ripe) = 0.6 × 0.0003777 = 0.0002266 and P(Unripe)×P(175|Unripe) = 0.4 × 0.0017532 = 0.0007013. Their sum is 0.0009279. Dividing each score by that sum:

P(Ripe|175g)   ≈ 0.0002266 / 0.0009279 ≈ 0.244  (24.4%)
P(Unripe|175g) ≈ 0.0007013 / 0.0009279 ≈ 0.756  (75.6%)

Despite Ripe being the more common class overall (60% prior), and despite 175 g sitting exactly halfway between the two means, this specific mango is classified as Unripe with 75.6% confidence — because the evidence (the wide Unripe spread) outweighs the class imbalance in the prior. This is the prior-versus-likelihood tug-of-war playing out with real numbers.

Worked Example 2 — Multinomial Naive Bayes: Spam Detection and the Zero-Frequency Problem

Text isn't continuous, so we don't use a Gaussian for it. Multinomial Naive Bayes instead estimates P(word | class) directly as a relative frequency: how often does this word show up among all the words ever seen in that class's training documents?

Suppose our entire training set is six short messages, three labelled spam and three labelled ham (legitimate):

• Spam: "win money win prize", "money free win", "prize free money"
• Ham: "meeting schedule money", "project update meeting", "schedule project team"

Counting words per class: Spam has 10 total word-tokens (win: 3, money: 3, free: 2, prize: 2). Ham has 9 total word-tokens (meeting: 2, schedule: 2, project: 2, money: 1, update: 1, team: 1). The full vocabulary across both classes has 9 distinct words, so |V| = 9. Priors: P(Spam) = P(Ham) = 3/6 = 0.5.

Now classify a new two-word message: "money prize". A first instinct is P(money|Spam) = 3/10 and P(prize|Spam) = 2/10. But look at "prize" in Ham: it appears zero times in the ham training documents. A raw frequency estimate gives P(prize|Ham) = 0/9 = 0. Because Naive Bayes multiplies likelihoods together, a single zero anywhere in the product forces the entire class score to zero — no matter how strongly every other word points to Ham. One unseen word would make Ham mathematically impossible for this message, which is clearly too extreme a conclusion to draw from the mere absence of one word in a small training set.

This is the zero-frequency problem, and the fix is Laplace (add-one) smoothing: pretend every word in the vocabulary was seen one extra time in every class, before counting. The estimator becomes

P(word|class) = (count(word,class) + 1) / (total_words_in_class + |V|)

Adding 1 to every count and |V| to every denominator keeps every probability strictly positive without needing to see every word in every class, and as training data grows the smoothing's effect shrinks to nothing. Recomputing with smoothing (|V| = 9):

P(money|Spam) = (3+1)/(10+9) = 4/19 ≈ 0.21053
P(prize|Spam) = (2+1)/(10+9) = 3/19 ≈ 0.15789
P(money|Ham)  = (1+1)/(9+9)  = 2/18 ≈ 0.11111
P(prize|Ham)  = (0+1)/(9+9)  = 1/18 ≈ 0.05556

Unnormalized posterior scores (prior × product of word likelihoods):

Spam: 0.5 × 0.21053 × 0.15789 = 0.5 × (4/19)(3/19) = 6/361 ≈ 0.016620
Ham:  0.5 × 0.11111 × 0.05556 = 0.5 × (2/18)(1/18) = 1/324 ≈ 0.003086

Normalizing (sum = 0.016620 + 0.003086 = 0.019707):

P(Spam|"money prize") ≈ 0.016620/0.019707 ≈ 0.8434  (84.3%)
P(Ham |"money prize") ≈ 0.003086/0.019707 ≈ 0.1566  (15.7%)

The message classifies as Spam with about 84% confidence. Note that Ham's score is no longer zero, but it's still small — smoothing prevented an impossible conclusion without pretending "prize" is somehow common in legitimate mail.

Here is the same computation as working code, so you can trace exactly how the hand calculation above turns into a program:

from collections import defaultdict

spam_docs = ["win money win prize", "money free win", "prize free money"]
ham_docs  = ["meeting schedule money", "project update meeting", "schedule project team"]

def word_counts(docs):
    counts = defaultdict(int)
    total = 0
    for doc in docs:
        for word in doc.split():
            counts[word] += 1
            total += 1
    return counts, total

spam_counts, spam_total = word_counts(spam_docs)
ham_counts, ham_total = word_counts(ham_docs)
vocab = set(spam_counts) | set(ham_counts)
V = len(vocab)

def word_prob(word, counts, total, V):
    return (counts.get(word, 0) + 1) / (total + V)

def classify(message, prior_spam=0.5, prior_ham=0.5):
    p_spam, p_ham = prior_spam, prior_ham
    for w in message.split():
        p_spam *= word_prob(w, spam_counts, spam_total, V)
        p_ham  *= word_prob(w, ham_counts, ham_total, V)
    total = p_spam + p_ham
    return p_spam / total, p_ham / total, p_spam, p_ham

norm_spam, norm_ham, raw_spam, raw_ham = classify("money prize")
print(round(raw_spam, 6), round(raw_ham, 6))
print(round(norm_spam, 4), round(norm_ham, 4))

Tracing it: spam_total = 10, ham_total = 9, V = 9 (the nine distinct words listed earlier). word_prob("money", spam_counts, 10, 9) = (3+1)/(10+9) = 4/19, matching the hand calculation exactly. The function multiplies these across both words of "money prize" for each class, then normalizes. Running it prints 0.01662 0.003086 on the first line (the raw, unnormalized scores) and 0.8434 0.1566 on the second (the normalized posterior probabilities) — identical to the values derived by hand above, because the code performs the exact same arithmetic, just without rounding at each intermediate step.

Common Misconception: "If the Classification Is Right, the Probability Must Be Accurate"

A very natural but incorrect belief is that because Naive Bayes classifies correctly so often, the actual probability numbers it reports — like the 84.3% above — must be trustworthy, well-calibrated estimates of true confidence. They usually are not. The independence assumption, when violated (as it almost always is — "free" and "prize" really do tend to appear together in spam, which is a dependency the model ignores), tends to make Naive Bayes's probability estimates overconfident: scores get pushed toward 0 or 1 more aggressively than the real-world probabilities justify.

Why, then, does the classifier still pick the right class so often? Pedro Domingos and Michael Pazzani addressed exactly this puzzle in their 1997 paper "On the Optimality of the Simple Bayesian Classifier under Zero-One Loss" (Machine Learning, 29, 103–130). Their key insight: classification accuracy under zero-one loss (simply "did you pick the right class or not") only requires the correct class to have the highest score — it does not require the score itself to be numerically accurate. They showed that violations of the independence assumption often distort every class's score in a similar direction and by a similar amount, so the ranking between classes — which one comes out on top — survives even when the individual numbers are badly skewed. The lesson: trust Naive Bayes's decisions more than you trust its stated confidence levels. If you need calibrated probabilities (say, to rank how urgently to review flagged transactions), that number needs separate calibration, not the raw Naive Bayes output.

Where This Shows Up in Your Exams

Conditional probability and Bayes' theorem are core CBSE Class 12 Probability syllabus (the section-pass-rate style problem worked above is exactly that format), and both IIT-JEE (Main and Advanced) and BITSAT test them directly, typically as "given this happened, find the probability it came from this source" word problems built on the law of total probability plus Bayes' theorem, precisely as derived in this chapter. Olympiad tracks such as RMO and INMO occasionally push the same idea further into combinatorics-heavy conditional-probability problems, where you first have to count outcomes carefully before Bayes' theorem is even applicable. If you go on to a computer science degree, this exact algorithm — under the name Naive Bayes — appears as a standard supervised-learning topic in GATE's Data Science and Artificial Intelligence paper, usually alongside k-nearest neighbours and decision trees as a first family of classifiers to compare.

Active Recall — Test Yourself

Q1. Why is the classifier called "naive," and what specifically does the assumption claim about the features?

Answer: It assumes every feature is conditionally independent of every other feature, given the class — i.e. P(xi | x1,...,xn, C) = P(xi | C) for each feature. This is "naive" because it is almost always false (features like co-occurring words, or correlated measurements, are rarely truly independent), yet assuming it collapses an exponential number of joint parameters into a linear number of per-feature parameters, making the model tractable to train from limited data.

Q2. A market has two mango crates. Crate P holds 60 ripe and 40 unripe mangoes (100 total). Crate Q holds 20 ripe and 80 unripe mangoes (100 total). A crate is chosen at random with equal probability, then a mango is drawn from it and found to be ripe. What is the probability it came from Crate P?

Answer: P(P) = P(Q) = 0.5. P(Ripe|P) = 60/100 = 0.6, P(Ripe|Q) = 20/100 = 0.2. Total probability: P(Ripe) = 0.5(0.6) + 0.5(0.2) = 0.3 + 0.1 = 0.4. Bayes' theorem: P(P|Ripe) = P(Ripe|P)P(P)/P(Ripe) = (0.6)(0.5)/0.4 = 0.3/0.4 = 0.75, so 75%.

Q3. Using the mango Gaussian classifier from Worked Example 1 (Ripe: μ=200, σ=8; Unripe: μ=150, σ=10; priors 0.6/0.4), classify a mango weighing 190 g.

Answer: Ripe: (190-200)²=100, exponent =100/128=0.78125, exp(-0.78125)≈0.45784, P(190|Ripe) ≈ 0.04987 × 0.45784 ≈ 0.022830. Unripe: (190-150)²=1600, exponent =1600/200=8, exp(-8)≈0.0003355, P(190|Unripe) ≈ 0.03989 × 0.0003355 ≈ 0.0000134. Scores: Ripe 0.6×0.022830=0.013698; Unripe 0.4×0.0000134=0.0000054. Normalizing, P(Ripe|190g) ≈ 0.9996 (99.96%) — essentially certain, because 190 g is close to Ripe's mean and far outside Unripe's narrow-tailed range, unlike the 175 g case which was a genuine toss-up.

Q4. Using the spam/ham word model from Worked Example 2, classify the message "win schedule" (one word typical of spam, one typical of ham).

Answer: P(win|Spam)=(3+1)/19=4/19; P(schedule|Spam)=(0+1)/19=1/19 (schedule never appears in spam training docs). P(win|Ham)=(0+1)/18=1/18 (win never appears in ham); P(schedule|Ham)=(2+1)/18=3/18. Spam score: 0.5 × (4/19)(1/19) = 0.5 × 4/361 = 2/361 ≈ 0.005540. Ham score: 0.5 × (1/18)(3/18) = 0.5 × 3/324 = 1.5/324 ≈ 0.004630. Normalizing (sum ≈ 0.010170): P(Spam) ≈ 0.545 (54.5%), P(Ham) ≈ 0.455 (45.5%) — a genuinely close call, because the two words pull in opposite directions almost equally.

Summary

Naive Bayes turns classification into an application of one theorem, derived here from nothing more than the definition of conditional probability: P(C|X) ∝ P(X|C)·P(C), prior times likelihood, normalized across classes. The "naive" conditional-independence assumption is what makes the likelihood computable at all — turning an exponential joint distribution into a simple product of per-feature terms — at the cost of that assumption almost never being literally true. For continuous features, Gaussian Naive Bayes models each class as a bell curve and reads off likelihood from the Gaussian PDF, as in the mango example, where a wider spread let the Unripe class outscore Ripe even at equal distance from both means. For discrete/count features like words, Multinomial Naive Bayes estimates likelihoods from relative frequency, and needs Laplace smoothing to survive words that were never seen in one of the training classes without collapsing that class's entire score to zero. And per Domingos and Pazzani's 1997 result, the classifier's raw probability outputs can be badly miscalibrated even when its actual class decisions are reliably correct — because correct classification only needs the right class to win, not the right numbers to appear.

Think About It

Think about this: How would you explain naive bayes: probabilistic classification to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where naive bayes: probabilistic classification is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting naive bayes: probabilistic classification to at least 3 other topics you have studied.