Text Classification: Categorizing Documents

Open your phone's SMS app. Somewhere in the last week you probably got a message from your bank confirming a UPI payment, an IRCTC message with your PNR status, a "Congratulations! You've won a lucky draw, click here" spam text, and a friend's message about tonight's cricket score. Your phone silently sorted some of these into a Spam folder and left the rest in your main inbox. It did this without a human reading each message and without anyone hand-writing a rule for every possible spam sentence. Something computed a decision from the words in the text and put the message in a category. That "something" is text classification, and by the end of this chapter you will be able to build one — with actual numbers, not just a diagram — for a two-class problem, by hand and in code.

What "Classification" Actually Means

Formally, text classification is a function. You are given a fixed, finite set of classes (also called labels or categories) — for example {Spam, Not Spam} or {Sports, Politics, Technology}. You are given a document, which is just a string of text. The task is to build a function f that takes the document as input and outputs exactly one class from the set:

f(document) → class, where class ∈ {C₁, C₂, ..., Cₖ}

This looks trivial written this way, but the hard part is that f has to be learned from examples, not hand-coded. You cannot write "if the message contains the word 'lottery', it's spam" as a permanent rule — spammers change wording, and legitimate messages sometimes contain words like "prize" (a cricket match prize) or "won" (India won the match) without being spam at all. A classifier has to learn, from a labelled training set, which words are statistically associated with which class, and then combine that evidence for a brand-new document it has never seen. Everything in this chapter builds toward exactly that combination step.

Before any of that can happen, though, text — which is unstructured — has to be converted into numbers, because every classification algorithm operates on numbers, not on strings.

Step 1: Turning Words Into Numbers — Bag of Words

The simplest way to turn a document into numbers is to treat it as an unordered collection of words and count how often each word occurs. This is called the Bag of Words (BoW) model, because — like items dumped into a bag — the order of the words is thrown away; only the counts survive.

Before counting, real pipelines apply text normalization: lowercase everything so "India" and "india" are treated as the same token, split the text into individual words (tokenization), and usually remove stopwords — extremely common words like "the", "is", "a", "in" that carry almost no information about the topic of a document and would otherwise dominate every count.

Take three short headlines as a toy corpus:

• Doc1: "india wins cricket match"
• Doc2: "india wins election debate"
• Doc3: "kohli scores century"

After normalization the vocabulary — the set of all distinct words across the whole corpus — is: india, wins, cricket, match, election, debate, kohli, scores, century. That is 9 unique words, so each document becomes a 9-dimensional count vector. Laid out as a term-document matrix (one row per document, one count per vocabulary word), this corpus looks like:

• Doc1 → india:1, wins:1, cricket:1, match:1, election:0, debate:0, kohli:0, scores:0, century:0
• Doc2 → india:1, wins:1, cricket:0, match:0, election:1, debate:1, kohli:0, scores:0, century:0
• Doc3 → india:0, wins:0, cricket:0, match:0, election:0, debate:0, kohli:1, scores:1, century:1

This is exactly the "Term Document Matrix" you will see named in CBSE's Artificial Intelligence NLP unit. Notice already that Doc1 and Doc2 share two words (india, wins) — a hint that raw shared words are the signal a classifier will exploit — while Doc3 shares nothing with either, making it look topically distant from both.

Step 2: Not All Words Deserve Equal Weight — TF-IDF

Raw counts have a serious flaw: a word that appears in almost every document in the corpus (imagine "india" appearing in 900 of 1000 news headlines) tells you almost nothing about what makes one document different from another, yet it can still get a large raw count. What you actually want is a weight that rewards a word for being frequent in this document but penalizes it for being common across the whole corpus. That is precisely what Term Frequency–Inverse Document Frequency (TF-IDF) computes.

Term Frequency for a word t in document d is its relative frequency within that document:

TF(t, d) = (number of times t appears in d) / (total number of words in d)

Document Frequency, DF(t), is the number of documents in the corpus that contain t at least once (not how many times — just whether it appears). Inverse Document Frequency then inverts and compresses that count logarithmically:

IDF(t) = log(N / DF(t)), where N is the total number of documents in the corpus

The logarithm matters, not just decoration: without it, a word appearing in 1 of 1000 documents would get a weight 1000× larger than a word appearing in all 1000, wildly overpowering everything else. The log compresses that gap to a manageable, smoothly varying scale while preserving the ordering — rarer words still get higher IDF, just not explosively so. Finally:

TF-IDF(t, d) = TF(t, d) × IDF(t)

Apply this to the 3-document corpus above, using log base 10 and N = 3. The word "india" appears in Doc1 and Doc2, so DF(india) = 2, giving IDF(india) = log₁₀(3/2) = log₁₀(1.5) ≈ 0.1761. The word "kohli" appears only in Doc3, so DF(kohli) = 1, giving IDF(kohli) = log₁₀(3/1) = log₁₀(3) ≈ 0.4771 — nearly three times the weight, purely because it is rarer across the corpus.

Now compute the full TF-IDF score for "india" in Doc1 (4 words total, "india" appears once): TF = 1/4 = 0.25, so TF-IDF(india, Doc1) = 0.25 × 0.1761 ≈ 0.0440. Compare "kohli" in Doc3 (3 words total, "kohli" appears once): TF = 1/3 ≈ 0.3333, so TF-IDF(kohli, Doc3) = 0.3333 × 0.4771 ≈ 0.1590 — more than three times larger, even though both words occur exactly once in their document. TF-IDF has correctly identified that "kohli" is the more distinctive, topic-defining word for Doc3, while "india" is diluted because it is shared across the corpus. This weighting is exactly why TF-IDF vectors, rather than raw Bag-of-Words counts, feed most real-world text classifiers.

You can verify the preprocessing step in code before any weighting is applied:

from collections import Counter

def tokenize(text):
    stopwords = {"the", "is", "a", "in", "on", "of", "to"}
    words = text.lower().split()
    return [w for w in words if w not in stopwords]

doc = "India wins the election"
tokens = tokenize(doc)
bow = Counter(tokens)
print(tokens)
print(bow)

Trace it by hand: doc.lower() gives "india wins the election"; .split() gives ["india", "wins", "the", "election"]; the list comprehension drops "the" because it is in stopwords, leaving ["india", "wins", "election"]. Counter then counts each token once. The output is exactly:

['india', 'wins', 'election']
Counter({'india': 1, 'wins': 1, 'election': 1})

Step 3: The Classifier — Naive Bayes

Vectors alone do not classify anything; you need a rule that maps a vector to a class. The classical, and still genuinely important, algorithm for this is Naive Bayes. It is built directly on Bayes' theorem, which you will also meet in NCERT Class 12 probability: for events A and B, the definition of conditional probability gives P(A|B) = P(A∩B)/P(B) and P(B|A) = P(A∩B)/P(A). Since both equal P(A∩B), setting them equal and rearranging gives:

P(A|B) = [P(B|A) × P(A)] / P(B)

Substitute A = "class" and B = "document" and you get exactly what a classifier needs — the probability of a class given the observed document:

P(class | document) = [P(document | class) × P(class)] / P(document)

To pick the best class, you compute this for every candidate class and take the one with the highest value — this is called the argmax decision rule. Crucially, P(document) in the denominator is the same number regardless of which class you are testing, so it never changes which class wins; you can drop it entirely and just compare P(document | class) × P(class) across classes.

The remaining difficulty is P(document | class): a document is a sequence of words w₁, w₂, ..., wₙ, and computing the true joint probability P(w₁, w₂, ..., wₙ | class) would require accounting for every dependency between every pair of words — computationally and statistically infeasible from a small training set. Naive Bayes makes a simplifying assumption: it treats every word as conditionally independent of every other word, given the class. That collapses the joint probability into a simple product:

P(document | class) ≈ P(w₁|class) × P(w₂|class) × ... × P(wₙ|class)

This assumption is why the algorithm is called "naive" — words are obviously not really independent (the word "not" changes the meaning of whatever follows it, and "election" is more likely right after "Lok Sabha" than after a random word). The assumption is literally false. But here is the important, often-missed nuance: Naive Bayes does not need P(document|class) to be numerically accurate — it only needs the class with the largest score to be correctly identified. In practice, even with the independence assumption violated, the ranking between classes frequently still comes out right, which is why the algorithm remains a strong, fast baseline despite its "naive" simplification.

One more real problem remains. If a word in the test document never appeared in the training data for some class, its estimated probability P(word|class) would be exactly 0/total = 0. Because the classifier multiplies all the word probabilities together, a single unseen word would force the entire product to zero, wiping out the influence of every other word in the document — clearly too fragile. The fix is Laplace (add-one) smoothing: add 1 to every word's count and add the vocabulary size |V| to the denominator, so no word ever gets exactly zero probability:

P(word | class) = [count(word, class) + 1] / [total words in class + |V|]

This preserves a valid probability distribution — summing (count+1)/(total+|V|) over every word in the vocabulary gives (total + |V|)/(total + |V|) = 1, exactly as a probability distribution must.

Worked Example: Classifying "India wins the election"

Build a two-class training set — Sports vs Politics — with three documents each (after stopword removal):

• Sports: "india wins cricket match", "kohli scores century today", "india beats australia final"
• Politics: "parliament passes new bill", "minister wins election debate", "government announces new policy today"

Sports has 12 word tokens total, with india appearing 2 times and wins appearing 1 time. Politics has 13 word tokens total, with india appearing 0 times, wins 1 time, and election 1 time. The combined vocabulary across both classes has 21 distinct words. Both classes have 3 of the 6 total training documents, so the prior is P(Sports) = P(Politics) = 0.5.

The test document, after normalization, is [india, wins, election]. Apply Laplace-smoothed Naive Bayes for Sports (total = 12, |V| = 21, denominator = 33):

• P(india|Sports) = (2+1)/33 = 3/33 ≈ 0.0909
• P(wins|Sports) = (1+1)/33 = 2/33 ≈ 0.0606
• P(election|Sports) = (0+1)/33 = 1/33 ≈ 0.0303 — note election never occurred in Sports training data, yet smoothing keeps this nonzero instead of zeroing out the whole product

And for Politics (total = 13, |V| = 21, denominator = 34):

• P(india|Politics) = (0+1)/34 = 1/34 ≈ 0.0294
• P(wins|Politics) = (1+1)/34 = 2/34 ≈ 0.0588
• P(election|Politics) = (1+1)/34 = 2/34 ≈ 0.0588

Multiplying three probabilities under 0.1 produces very small numbers (roughly 0.0002 and 0.0001), and real classifiers deal with far longer documents than three words — multiplying hundreds of small fractions would underflow a computer's floating-point precision to exactly 0. The standard fix is to take logarithms: since log is a strictly increasing function, comparing log-scores gives the identical ranking as comparing the original probabilities, while turning products into sums that never underflow.

Score(Sports) = log P(Sports) + log P(india|Sports) + log P(wins|Sports) + log P(election|Sports)
= log₁₀(0.5) + log₁₀(0.0909) + log₁₀(0.0606) + log₁₀(0.0303)
= -0.3010 + (-1.0414) + (-1.2175) + (-1.5185) = -4.0784

Score(Politics) = log₁₀(0.5) + log₁₀(0.0294) + log₁₀(0.0588) + log₁₀(0.0588)
= -0.3010 + (-1.5315) + (-1.2304) + (-1.2304) = -4.2934

Since -4.0784 is greater (less negative) than -4.2934, the classifier predicts Sports. Notice what happened: "election" on its own leans strongly toward Politics, but "india" appearing twice in the Sports training data pulled the combined score the other way. This is the independence assumption in action — every word votes independently, and the class with the strongest combined vote wins, even when one individual word disagrees.

You can check this arithmetic directly in code:

import math

vocab_size = 21
sports_counts = {"india": 2, "wins": 1, "election": 0}
politics_counts = {"india": 0, "wins": 1, "election": 1}
sports_total = 12
politics_total = 13

def word_prob(word, counts, total):
    return (counts.get(word, 0) + 1) / (total + vocab_size)

test_words = ["india", "wins", "election"]
log_sports = math.log10(0.5)
log_politics = math.log10(0.5)

for w in test_words:
    log_sports += math.log10(word_prob(w, sports_counts, sports_total))
    log_politics += math.log10(word_prob(w, politics_counts, politics_total))

print(round(log_sports, 4), round(log_politics, 4))

Tracing it: word_prob("india", sports_counts, 12) = (2+1)/(12+21) = 3/33 = 0.09091, and the same pattern for the rest, exactly matching the hand computation above. The final print statement outputs -4.0784 -4.2934, confirming Sports wins.

Common Misconception: "High Accuracy Means a Good Classifier"

Once a classifier makes predictions, you need to measure how good it actually is — and the most intuitive-seeming metric, accuracy, is frequently misleading, especially for exactly the kind of task this chapter opened with: spam detection, where most real messages are not spam.

Suppose you test a spam filter on 100 SMS messages, of which 10 are truly spam and 90 are truly legitimate (ham). The filter correctly flags 6 of the 10 spam messages (these are True Positives, TP = 6) but misses 4, letting them through as ham (False Negatives, FN = 4). Of the 90 legitimate messages, it correctly leaves 85 alone (True Negatives, TN = 85) but wrongly flags 5 as spam (False Positives, FP = 5). Lay these four numbers out as a confusion matrix — predicted spam vs predicted ham, against actual spam vs actual ham — and every classification-quality metric is computed directly from it:

• Accuracy = (TP + TN) / Total = (6 + 85) / 100 = 91%
• Precision = TP / (TP + FP) = 6 / 11 ≈ 54.5% — of the messages the filter called spam, how many really were spam
• Recall = TP / (TP + FN) = 6 / 10 = 60% — of the messages that really were spam, how many did the filter catch
• F1-score = 2 × (Precision × Recall) / (Precision + Recall) = 2 × (0.545 × 0.6) / (0.545 + 0.6) ≈ 57.1% — the harmonic mean, which only stays high when both precision and recall are reasonably high

Here is the misconception, made concrete: imagine a lazy "classifier" that ignores the text entirely and labels every single message as "ham." On this same test set it would score Accuracy = 90/100 = 90% — nearly identical to the real filter's 91%. Yet this lazy classifier has TP = 0 and Recall = 0%: it catches literally zero spam messages, which is precisely the one job a spam filter exists to do. Accuracy looked almost as good only because the classes are imbalanced (90 ham vs 10 spam) — predicting the majority class by default is a cheap way to inflate accuracy without learning anything. This is why precision, recall, and F1 — not accuracy alone — are the metrics actually reported for classification tasks with imbalanced classes, and why you should be suspicious of any classifier whose accuracy is quoted without its confusion matrix.

A second common misconception is about Bag of Words itself: because it discards word order, "india wins match" and "match wins india" produce the identical count vector, and BoW cannot by itself distinguish "not good" from "good" — negation is invisible to a model that only counts individual words. This is a genuine limitation, not a bug to be argued away; it is one reason more advanced NLP methods (word n-grams, and eventually sequence-aware neural models, which are beyond this chapter) exist. For the two-class, short-document problems this chapter covers, BoW and TF-IDF combined with Naive Bayes remain a fast, interpretable, and surprisingly competitive baseline — which is exactly why it is still taught as the entry point to statistical NLP.

Visualizing the Full Pipeline

Where This Fits: CBSE and Competitive Exams

CBSE's Artificial Intelligence curriculum's statistical NLP unit is built almost entirely around the sequence this chapter followed: text normalization, Bag of Words, Term Frequency, Document Frequency, and TF-IDF, culminating in a Term-Document Matrix — the exact terminology and formulas used above are the ones you are expected to reproduce in board examination and practical file questions. The classifier layer goes a step further than the board syllabus and connects directly to your Class 12 Mathematics probability chapter: Bayes' theorem, conditional probability, and the total probability theorem are core JEE Main topics, and Naive Bayes is nothing more than that same theorem applied to word counts instead of urns and coins — practising Bayes' theorem numericals for JEE strengthens the exact reasoning used here, and vice versa. At the undergraduate and GATE level (particularly the GATE Data Science & AI paper, and the probability sections of GATE CS), Bayes classifiers, confusion matrices, precision, recall, and F1-score reappear as standard machine-learning syllabus items — so the confusion-matrix arithmetic in this chapter is not a one-time exercise but a metric vocabulary you will keep using.

Summary

Text classification learns a function from documents to a fixed set of classes. Since algorithms need numbers, documents are first converted to vectors: Bag of Words counts raw word frequencies in a term-document matrix, and TF-IDF re-weights those counts by term frequency divided by document frequency (in log form), so words that are frequent in one document but rare across the corpus get boosted while corpus-wide common words get suppressed. Naive Bayes classifies a vector by applying Bayes' theorem, assuming word probabilities are conditionally independent given the class (the "naive" step) so the joint likelihood becomes a simple product; Laplace smoothing adds one to every count to prevent an unseen word from zeroing out that product, and log-probabilities turn the product into a numerically stable sum without changing which class wins. Finally, a classifier's quality is judged with a confusion matrix — precision, recall, and F1-score — because accuracy alone is easily inflated by class imbalance and can hide a classifier that never catches the minority class it was built to catch.

Practice

• Using the Sports/Politics training data in the worked example, classify the test sentence "kohli wins the debate" (after stopword removal: [kohli, wins, debate]) by hand using Laplace-smoothed Naive Bayes with |V| = 21. Show both log-scores to 4 decimal places and state the predicted class.
• A news classifier is tested on 200 articles: 40 are truly Technology and 160 are truly Non-Technology. It predicts 25 of the 40 Technology articles correctly, misses 15, and among the 160 Non-Technology articles it wrongly labels 10 as Technology. Construct the full confusion matrix (TP, FP, FN, TN) and compute accuracy, precision, recall, and F1-score.
• For the 3-document corpus in the TF-IDF section, compute TF-IDF(cricket, Doc1) and TF-IDF(match, Doc1), and explain in one sentence why they equal each other numerically.
• Explain, using the independence assumption, why Naive Bayes can classify a document correctly even when one individual word in it points toward the wrong class.