Naive Bayes for Text Classification: Spam, Sentiment, and Language Detection

The SMS That Almost Fooled You

Your phone buzzes: "Dear customer, your UPI a/c will be BLOCKED in 24 hrs. Update KYC now: bit.ly/xyz123 to avoid suspension." Some part of your brain flags this instantly, even before you consciously reason about it. But how? You've never seen this exact sentence before. No human sat down and taught you a rule that says "if a message contains the word KYC and a shortened link, it is fraud." Instead, your brain is doing something statistical: certain words — "blocked," "urgent," "click," "update KYC now" — have shown up disproportionately often in messages you've learned to distrust, while words like "meet you at 6," "assignment," or "reached home" show up in messages from real people. You are, without writing any formula, running a rough probability calculation in your head.

Naive Bayes text classification turns that intuition into an exact, computable algorithm. Given a message, it asks: "Based on how often each word appears in spam messages versus genuine messages in a training set, which class is this new message more likely to belong to?" It is one of the oldest machine learning algorithms still in daily production use — email providers, SMS spam filters, and even some early language-identification tools run on exactly this math, or a close variant of it. This chapter builds it from the ground up: from the definition of conditional probability, through a full worked numerical example you can check by hand, to a working Python implementation, and finally to two more applications — sentiment analysis and language detection — that use the identical mathematical machinery with the features swapped out.

From Conditional Probability to Bayes' Theorem

Everything here rests on one definition. For two events A and B, with P(B) > 0, the conditional probability of A given B is defined as:

P(A | B) = P(A ∩ B) / P(B)

This is not an assumption — it is literally how "probability of A, given that B has happened" is defined: the fraction of B's outcomes that also satisfy A. By the identical logic, swapping the roles of A and B:

P(B | A) = P(A ∩ B) / P(A)

Both expressions share the same numerator, P(A ∩ B). Rearranging each definition to isolate it gives:

P(A ∩ B) = P(A | B) · P(B) = P(B | A) · P(A)

Taking the last two expressions and dividing both sides by P(B), we get Bayes' theorem:

P(A | B) = [ P(B | A) · P(A) ] / P(B)

That's the entire derivation — two rearrangements of one definition. You will meet this again, stated formally, in the Probability chapter of Class 12 Mathematics (conditional probability and Bayes' theorem, NCERT Part II). What we do next is decide what A and B should mean for a text classifier.

Turning Bayes' Theorem into a Classifier

Let A be "the message belongs to class c" (spam or ham — "ham" is the standard term for legitimate, non-spam mail), and let B be "the message is the exact sequence of words w₁, w₂, …, wₙ we observed," which we'll call the document d. Substituting into Bayes' theorem:

P(c | d) = [ P(d | c) · P(c) ] / P(d)

Read left to right: P(c | d) is the posterior — what we actually want, the probability the message is spam given the words it contains. P(c) is the prior — how common spam is overall, before reading a single word. P(d | c) is the likelihood — how probable this exact sequence of words is, if we already know the class. P(d) is the probability of seeing this document at all, regardless of class.

Here is the first genuinely useful simplification. We are not trying to compute the actual value of P(c | d) — we only need to know which class, spam or ham, has the larger posterior. Since P(d) does not depend on c (it's the same number no matter which class we're testing), it cannot change which class wins. So the classification rule becomes an argmax over the numerator alone:

ĉ = argmax_c [ P(d | c) · P(c) ]

This is a genuine simplification, not a hack: we've reduced "compute an exact probability" (hard — P(d) requires summing over every possible document) to "compare two numbers and pick the larger" (easy).

The "Naive" Assumption

One problem remains: P(d | c), the probability of an entire specific sentence given the class, is still intractable to estimate directly. A message with 10 words drawn from a vocabulary of even a few thousand words has astronomically many possible orderings; no training set is remotely large enough to have seen this exact sentence before, let alone enough times to estimate its probability reliably.

Naive Bayes solves this with the assumption that gives the algorithm its name: treat the document as a bag of words — an unordered collection — and assume each word's presence is conditionally independent of every other word, given the class. Formally:

P(d | c) = P(w₁, w₂, …, wₙ | c) ≈ P(w₁ | c) · P(w₂ | c) · … · P(wₙ | c) = ∏_i=1ⁿ P(wᵢ | c)

This assumption is false, and obviously so. In the phrase "not good," the word "not" completely flips the meaning contributed by "good" — the two words are deeply correlated, not independent. Word order and co-occurrence carry real information that a bag-of-words model throws away entirely. This is the "naive" in Naive Bayes: it is a known, named simplification, not an oversight. We accept it because it converts an intractable joint-probability estimation problem into n separate, easy, single-word estimation problems — and, as you'll see later in this chapter, the resulting classifier still performs remarkably well despite the assumption being technically wrong.

Combining this with the argmax rule from the previous section gives the full multinomial Naive Bayes decision rule:

ĉ = argmax_c [ P(c) · ∏_i=1ⁿ P(wᵢ | c) ]

Estimating the Probabilities: Counting, With a Fix

Everything above is theory. To actually classify a message you need numbers for P(c) and P(w | c), estimated from a labelled training set. The prior is the easiest: if your training set has 3 spam messages and 3 ham messages out of 6 total, P(spam) = 3/6 = 0.5 and P(ham) = 3/6 = 0.5 — just the fraction of training documents in each class.

For the word likelihood P(w | c), the natural estimate is relative frequency: out of every word token that appeared in class c's training documents, what fraction of them were the word w?

P(w | c) = count(w, c) / total_words_in_c

where count(w, c) is how many times word w appeared across all training documents of class c, and total_words_in_c is the total number of word tokens (with repetition) in class c's training documents. This is the maximum-likelihood estimate for a multinomial distribution — it is the single most "consistent with the data" choice of probabilities, in the precise sense that it's the estimate that makes the observed training counts the most probable outcome under a multinomial model.

There's a serious flaw in this raw formula, though. Suppose the word "lottery" never once appeared in your ham training messages. Then count("lottery", ham) = 0, so P("lottery" | ham) = 0. Now if a genuine test message happens to contain the word "lottery" even once — say, "my office lottery draw for Diwali gifts is today" — the entire product ∏ P(wᵢ | ham) becomes exactly zero, no matter how strongly every other word in the message points to "ham." One unseen word silences all the evidence from every other word. This is the zero-frequency problem, and it is a real, dependency-breaking bug in the naive formula, not a minor edge case — with a large vocabulary, some unseen word in a new test message is the norm, not the exception.

The standard fix is Laplace smoothing (also called add-one smoothing). Instead of the raw count, add 1 to every word's count before dividing, and compensate the denominator by adding the vocabulary size |V| (the number of distinct words across the whole training set):

P(w | c) = [ count(w, c) + 1 ] / [ total_words_in_c + |V| ]

Why exactly |V| in the denominator, and not some other number? Because probabilities over the vocabulary must sum to 1, and this specific choice is exactly what preserves that. Summing the smoothed formula over every word in the vocabulary:

Σ_w∈V [count(w,c)+1] / [total_words_in_c + |V|] = [Σ_wcount(w,c) + |V|] / [total_words_in_c + |V|] = [total_words_in_c + |V|] / [total_words_in_c + |V|] = 1

since Σ_w count(w,c) is, by definition, total_words_in_c (every token belongs to exactly one word type). So Laplace smoothing isn't an arbitrary patch — it's the minimal correction that both eliminates zero probabilities (every word now gets at least a small positive probability, count+1 ≥ 1) and keeps the estimate a valid probability distribution.

Worked Example: Classifying a New SMS by Hand

Now let's run the whole pipeline on real numbers you can verify with a calculator. Training set — 3 spam, 3 ham, each message already lowercased and split into words:

• Spam S1: "win money now"
• Spam S2: "win free money"
• Spam S3: "claim free prize now"
• Ham H1: "call you now"
• Ham H2: "meet me now"
• Ham H3: "call me tomorrow"

Step 1 — priors. 3 spam and 3 ham documents out of 6: P(spam) = P(ham) = 0.5.

Step 2 — word counts. Spam has 10 word tokens total (3+3+4), with counts win:2, money:2, now:2, free:2, claim:1, prize:1. Ham has 9 word tokens total (3+3+3), with counts call:2, you:1, now:2, meet:1, me:2, tomorrow:1.

Step 3 — vocabulary. The union of every distinct word across both classes is {win, money, now, free, claim, prize, call, you, meet, me, tomorrow} — exactly |V| = 11 words.

Step 4 — smoothed likelihoods for the test message "win free now". For spam, the denominator is total_words_in_spam + |V| = 10 + 11 = 21:

P(win|spam) = (2+1)/21 = 3/21 = 1/7 ≈ 0.1429
P(free|spam) = (2+1)/21 = 1/7 ≈ 0.1429
P(now|spam) = (2+1)/21 = 1/7 ≈ 0.1429

For ham, the denominator is 9 + 11 = 20. Note "win" and "free" never occurred in any ham training message, so their raw counts are 0 — this is exactly the zero-frequency case Laplace smoothing exists to handle:

P(win|ham) = (0+1)/20 = 0.05
P(free|ham) = (0+1)/20 = 0.05
P(now|ham) = (2+1)/20 = 3/20 = 0.15

Step 5 — combine.

Score(spam) = P(spam) · P(win|spam) · P(free|spam) · P(now|spam) = 0.5 × (1/7)³ = 0.5/343 ≈ 0.0014577

Score(ham) = P(ham) · P(win|ham) · P(free|ham) · P(now|ham) = 0.5 × 0.05 × 0.05 × 0.15 = 0.5 × 0.000375 = 0.0001875

Score(spam) ≈ 0.00146 is roughly 7.8 times larger than Score(ham) ≈ 0.00019, so the classifier confidently outputs spam — driven almost entirely by "win" and "free," words the ham class had never seen. Notice something instructive: for the word "now" alone, ham actually scores higher (0.15 vs 0.1429) — "now" alone is not evidence for spam. It's the joint weight of all three words together that tips the decision, exactly as the product rule intends.

Why We Use Log-Probabilities

In the hand example, multiplying just three probabilities already produced a number near 0.0001. Real messages have 15–30 words, and each per-word probability is well under 1. Multiplying thirty numbers each around 0.01–0.1 together can produce a result smaller than 10⁻³⁰, and computers store real numbers with finite precision (a standard 64-bit float underflows to exactly 0 somewhere around 10⁻³⁰⁸, but precision loss and comparison errors bite well before that). If two classes' true scores are both extremely small, floating-point rounding can make them indistinguishable or even flip which one looks larger.

The fix uses one property of the logarithm function: it is strictly increasing, so for positive x and y, x > y if and only if log(x) > log(y). Applying log to both sides never changes which class has the larger score — it only changes the numbers we compute with. Since log turns products into sums (log(a·b) = log(a) + log(b)), the decision rule becomes:

ĉ = argmax_c [ log P(c) + Σ_i=1ⁿ log P(wᵢ | c) ]

Summing thirty moderately-negative numbers is numerically stable in a way that multiplying thirty small positive fractions is not. Every production implementation of Naive Bayes uses log-probabilities internally for exactly this reason. Verifying on our worked example: log(spam score) = ln(0.5) + 3·ln(1/7) ≈ −0.6931 + 3(−1.9459) ≈ −6.5309, and log(ham score) = ln(0.5) + 2·ln(0.05) + ln(0.15) ≈ −0.6931 − 5.9915 − 1.8971 ≈ −8.5817. Since −6.5309 > −8.5817, spam wins — the identical answer as the raw-probability calculation, confirming the log transform preserves the decision, and e^(−6.5309−(−8.5817)) = e^2.0509 ≈ 7.77 matches the ratio 0.00146/0.00019 ≈ 7.77 computed directly above.

Implementing Multinomial Naive Bayes in Python

Here is the entire algorithm, trained and tested on the exact dataset from the worked example above, so you can trace every number:

from collections import defaultdict
import math

train_data = [
    (["win", "money", "now"], "spam"),
    (["win", "free", "money"], "spam"),
    (["claim", "free", "prize", "now"], "spam"),
    (["call", "you", "now"], "ham"),
    (["meet", "me", "now"], "ham"),
    (["call", "me", "tomorrow"], "ham"),
]

def train_naive_bayes(data):
    class_doc_count = defaultdict(int)
    word_count = defaultdict(lambda: defaultdict(int))
    total_words_in_class = defaultdict(int)
    vocab = set()

    for words, label in data:
        class_doc_count[label] += 1
        for w in words:
            word_count[label][w] += 1
            total_words_in_class[label] += 1
            vocab.add(w)

    total_docs = sum(class_doc_count.values())
    priors = {c: class_doc_count[c] / total_docs for c in class_doc_count}
    return priors, word_count, total_words_in_class, vocab

def word_prob(word, label, word_count, total_words_in_class, vocab):
    count = word_count[label][word]
    return (count + 1) / (total_words_in_class[label] + len(vocab))

def classify(words, priors, word_count, total_words_in_class, vocab):
    scores = {}
    for label in priors:
        log_score = math.log(priors[label])
        for w in words:
            p = word_prob(w, label, word_count, total_words_in_class, vocab)
            log_score += math.log(p)
        scores[label] = log_score
    best = max(scores, key=scores.get)
    return best, scores

priors, word_count, total_words_in_class, vocab = train_naive_bayes(train_data)
label, scores = classify(["win", "free", "now"], priors, word_count,
                          total_words_in_class, vocab)
print(label)
print(scores)

Tracing it: vocab ends up with 11 entries, matching Step 3. total_words_in_class["spam"] is 10 and total_words_in_class["ham"] is 9, matching Step 2. word_prob("win", "spam", ...) computes (2+1)/(10+11) = 1/7, matching Step 4 exactly. The final printed output is spam, followed by a dictionary whose "spam" entry is approximately −6.531 and whose "ham" entry is approximately −8.582 — identical to the hand-derived log-scores above. This is the entire algorithm; commercial spam filters add tokenization rules, stop-word handling, and much larger vocabularies, but the classification core is precisely this function.

Common Misconception: "The Independence Assumption Is Violated, So the Classifier Must Be Weak"

Students who correctly notice that word independence is false often draw the wrong conclusion: that Naive Bayes should therefore be a poor classifier compared to methods that model word dependencies. This is a reasonable-sounding inference, and it is wrong in practice far more often than intuition suggests.

The resolution is that classification and probability estimation are different goals with different success conditions. Naive Bayes only needs to get the ordering of the two class scores right — Score(spam) > Score(ham) — not the exact calibrated value of each probability. Pedro Domingos and Michael Pazzani's well-known 1997 analysis, "On the Optimality of the Simple Bayesian Classifier under Zero-One Loss," showed formally that Naive Bayes can classify optimally even when the independence assumption is badly violated, because dependencies between words often affect both classes' scores in similar, correlated ways — the errors introduced by the false assumption partially cancel out rather than accumulating into a wrong decision. This is precisely why Naive Bayes remains a genuinely competitive baseline for spam filtering and text classification even against far more sophisticated models: its bias is strong and wrong in detail, but the resulting decision boundary is often close enough to correct anyway, and it needs far less training data to get there than more flexible models do.

Beyond Spam: Sentiment Analysis with the Same Machinery

Nothing about the algorithm above is spam-specific — it is a general recipe: pick classes, count word frequencies per class in labelled training data, apply Laplace smoothing, and compare log-scores. For sentiment analysis on product reviews (the kind you'd find under a listing on Flipkart or Amazon), the classes become "positive" and "negative" instead of "spam" and "ham," and training documents are past reviews already labelled by star rating. Words like "excellent," "recommend," and "durable" accumulate high counts in the positive class; words like "waste," "defective," and "pathetic" accumulate high counts in the negative class. The exact same P(c) · ∏P(wᵢ|c) formula, the same Laplace-smoothing fix for words the model has never seen in one class, and the same log-sum computation apply unchanged — only the labels and the training corpus differ. This portability is the real payoff of having derived the algorithm from first principles rather than memorizing "the spam-filter algorithm": once you understand what the classes and the bag-of-words features represent, you can point it at any labelled text classification task.

Beyond Words: Language Detection with Character N-Grams

Language identification — deciding whether "yeh accha hai" is Hindi (in Roman script) or "this is good" is English — exposes a limitation of word-level features: short texts or informal, transliterated text (common across Indian social media and messaging, where Hindi, Tamil, or other languages get typed in Latin letters) often contain words the model has never seen at all, so word-level bag-of-words features run out of signal fast.

The fix is to change what counts as a "word" for the model — not the classification algorithm itself. Instead of splitting text into whitespace-separated words, split it into overlapping character n-grams (commonly bigrams — pairs — or trigrams of consecutive letters). "yeh accha hai" produces character bigrams like "ye," "eh," "h ," " a," "ac," "cc," "ch," "ha," "a ," and so on; "this is good" produces "th," "hi," "is," "s ," " i," "is," "s ," " g," "go," "oo," "od." Certain bigrams turn out to be far more common in Romanized Hindi than in English and vice versa, purely from differences in how the two languages sound and get spelled out. Train a separate multinomial Naive Bayes model per language, using character-bigram counts instead of word counts, apply the identical Laplace-smoothed likelihood formula, and the identical argmax-of-log-scores decision rule — and you get a working language identifier. This is not a hypothetical: several real, widely used open-source language-identification tools, including the once-popular langid.py library (Lui and Baldwin, 2012), are built on exactly a Naive Bayes classifier over character n-gram features. The lesson generalizes: Naive Bayes is really a template — choose a set of discrete features, assume they're conditionally independent given the class, and count — and the feature choice (words vs. character n-grams vs. anything else countable) is what adapts the same three equations to a new problem.

Where This Fits: CBSE, JEE, and Beyond

Bayes' theorem itself is formal Class 12 Mathematics content (Probability chapter, NCERT Part II) — this chapter gives you the reasoning and a concrete computational application well ahead of that, which should make the formal treatment easier to absorb, not harder, when you reach it. JEE Main and Advanced do not test machine learning directly, since they examine Physics, Chemistry, and Mathematics rather than applied Computer Science, so don't expect a Naive Bayes question on either paper. Where this content pays off directly is later: GATE's Computer Science and Data Science & AI syllabi explicitly list Bayes' theorem and Bayesian/probabilistic classifiers under machine learning, and any undergraduate course or competition problem involving probabilistic reasoning over counted events — a category Informatics Olympiad problems occasionally touch — draws on the same conditional-probability manipulation you derived in this chapter's second section. The lasting transferable skill is not "memorize the spam-filter formula" but "know how to turn P(A|B) into P(B|A) using nothing but the definition of conditional probability" — that manipulation reappears constantly, far beyond text classification.

Check Your Understanding

1. Using the training data from the worked example, compute P("claim" | spam) and P("claim" | ham) with Laplace smoothing, and state in one sentence why the ham probability is nonzero even though "claim" never appeared in any ham training message.
2. Classify the message "call me now" using the same trained model (priors, counts, and vocabulary from the worked example). Compute the smoothed likelihood for each of the three words under both classes, combine them with the priors, and state which class wins and by roughly what ratio.
3. Suppose you deliberately trained without any smoothing (using raw counts). Explain precisely which word in "call me now" would break the ham score, and why, referencing the zero-frequency problem.
4. A classmate says: "Since Naive Bayes assumes all words are independent, and that's clearly false for real sentences, it must always perform worse than a model that captures word order." Using the reasoning from the misconception section, explain what's wrong with this argument.
5. If you were building a language identifier for Hindi-in-Roman-script versus English SMS messages, explain why character bigrams are a better feature choice than whole words, and describe — in terms of counts and the smoothed-probability formula — how you would estimate P(bigram | language) from a training corpus.

Diagram: Smoothed Word Probabilities, Spam vs. Ham

Summary

Naive Bayes is built from exactly three ideas, stacked in order. First, Bayes' theorem — P(c|d) = P(d|c)P(c)/P(d) — derived from nothing more than the definition of conditional probability, lets us flip "probability of words given class" (estimable from training data) into "probability of class given words" (what we actually want), and since P(d) is constant across classes, classification reduces to argmax_c[P(c)·P(d|c)]. Second, the naive conditional-independence assumption, P(d|c) ≈ ∏P(wᵢ|c), trades a false-but-useful simplification for computational tractability, converting one impossible joint-probability estimate into n easy per-word estimates. Third, Laplace smoothing, P(w|c) = (count(w,c)+1)/(total_words_in_c+|V|), repairs the zero-frequency problem by guaranteeing every word gets nonzero probability while keeping the distribution valid (summing to 1). Log-probabilities make the resulting computation numerically stable without changing the decision, since log is monotonic. The same three-equation template — pick features, assume conditional independence given the class, smooth and count — powers spam filtering, sentiment analysis, and, with character n-grams substituted for words, language identification. The mathematics doesn't change between these applications; only what you choose to count does.

Think About It

Think about this: How would you explain naive bayes for text classification: spam, sentiment, and language detection to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.