Sentiment Analysis Pipeline: Building End-to-End

The Review That Fools a Simple Detector

Read this Flipkart review of a train berth cushion and decide, in your head, whether the customer is happy: "The seats were not comfortable, but the staff was extremely polite." Most humans read that as a mildly negative review — the actual product (the seat) failed, and the one nice thing mentioned is about a person, not the item being reviewed. Now watch what a naive "count the positive words" detector does with the same sentence. It scans for a dictionary of positive words and finds two: comfortable and polite. It scans for negative words and finds zero — "not" is not in most sentiment word-lists because "not" by itself carries no sentiment; it flips the sentiment of whatever comes after it. Positive count 2, negative count 0. Verdict: positive review. This is exactly backwards.

That single failure explains why "sentiment analysis" is not one function you call — it is a pipeline: a sequence of well-defined stages, each fixing a specific weakness of the stage before it, ending in a classifier that has actually learned the statistical relationship between words and sentiment rather than just counting them. This chapter builds that pipeline stage by stage, with every formula derived, not just quoted, and every claim checked against a worked numeric example you can reproduce by hand or in code.

The Full Pipeline, End to End

Every production sentiment system — whether it is Zomato scoring restaurant reviews, Amazon flagging fake 1-star ratings, or IRCTC's app-store review dashboard — runs the same five stages. The diagram below traces one review through all five, using the exact numbers you will derive by hand later in this chapter.

Stage 1 and 2: Tokenizing and Cleaning

Tokenization splits a string into the atomic units the model will reason about — usually words, sometimes sub-words. It sounds trivial until you hit "IRCTC's app," "10/10," or "😭." For this chapter we use whitespace tokenization after lowercasing, which is the same simplification CBSE-level NLP problems assume:

def tokenize(text):
    return text.lower().split()

print(tokenize("The App Crashed AGAIN!"))
# ['the', 'app', 'crashed', 'again!']

Notice "again!" still carries its punctuation — a real pipeline strips punctuation with a regular expression before splitting; we skip that detail here to keep the arithmetic in later sections clean, but you should know production tokenizers handle it. After tokenizing, stopword removal deletes high-frequency function words (a, the, is, was, and, but) that appear in almost every document regardless of sentiment and therefore add noise, not signal, to a small training set. Note the deliberate word "almost" — "but" and "not" must never be blindly removed, because they carry structural meaning about which sentiment word the writer actually intends. That single distinction is the seed of the misconception the next section corrects properly.

Misconception: "Sentiment Analysis Is Just Counting Positive and Negative Words"

This is the single most common wrong mental model, and the opening example was built specifically to break it. A word-counting approach treats every word as sentiment-independent — it never asks "positive about what, and under what condition?" Two structural patterns break pure word-counting, and a real pipeline must handle at least the first explicitly:

• Negation scope. "Not comfortable" is negative, but a bag of unigrams sees "not" and "comfortable" as two unrelated tokens. The standard fix, used in real preprocessing pipelines and simple enough to code by hand, is to tag every word between a negation cue and the next clause boundary with a NOT_ prefix, converting it into a new, distinct vocabulary token whose sentiment association the model can learn separately from the un-negated word.
• Contrastive conjunctions. "But," "however," and "although" signal that the second clause overrides the first in the writer's intended emphasis — this is why the pipeline's "but" resets negation scope in the code below, and why more advanced pipelines weight the post-"but" clause more heavily.

Here is a negation-scoping function that fixes exactly the seat-cushion review from the opening hook. Trace it by hand before running it:

NEGATIONS = {"not", "no", "never"}
STOPS = {".", ",", "but"}

def handle_negation(tokens):
    result = []
    negate = False
    for tok in tokens:
        if tok in STOPS:
            negate = False
            result.append(tok)
            continue
        if tok in NEGATIONS:
            negate = True
            result.append(tok)
            continue
        result.append("NOT_" + tok if negate else tok)
    return result

review = ["seats", "were", "not", "comfortable",
          "but", "staff", "was", "polite"]
print(handle_negation(review))

Trace it token by token. "seats," "were": negate is False, pass through unchanged. "not": matches NEGATIONS, so negate flips to True and "not" itself is appended as-is. "comfortable": negate is True, so it becomes NOT_comfortable — a brand-new vocabulary token, distinct from the plain "comfortable" that appears in genuinely positive reviews. "but": matches STOPS, so negate resets to False before "but" is appended — this ordering matters; if you reset negate after appending, "but" itself would incorrectly get prefixed. "staff," "was," "polite": negate is now False, so they pass through unchanged. Final output:

['seats', 'were', 'not', 'NOT_comfortable', 'but', 'staff', 'was', 'polite']

A classifier trained on data preprocessed this way learns that NOT_comfortable is a negative-leaning token, while plain "polite" stays a genuine positive signal — recovering the correct, mixed-but-net-negative reading a human gives that review, which pure word-counting could not.

Stage 3: Turning Words into Numbers

A classifier is a mathematical function; it cannot consume the string "crashed." Every word must become a number, and how you assign that number matters. The simplest scheme, bag-of-words, just counts how many times each vocabulary word appears in a document, ignoring order entirely — "app crashed" and "crashed app" produce the identical vector. This throws away word order (a real limitation, discussed later) but is enough to build a working classifier, which is why we use it for Stage 4's derivation.

Raw counts have a flaw: a word that appears in every review, positive or negative, tells you nothing about sentiment, yet a raw count treats it as equally important as a rare, highly discriminating word. TF-IDF (term frequency–inverse document frequency) fixes this by down-weighting words that are common across documents and up-weighting words that are distinctive to few. For a training set of N documents:

tf(word, doc)  = raw count of word in doc
df(word)       = number of documents containing word at least once
idf(word)      = ln( N / df(word) )
tfidf(word, doc) = tf(word, doc) × idf(word)

Work through a concrete case using four training reviews of the IRCTC train-booking app:

D1 (positive): "booking fast smooth"
D2 (positive): "app fast reliable"
D3 (negative): "booking failed slow"
D4 (negative): "app crashed slow"

N = 4. The word "fast" appears in D1 and D2, so df(fast) = 2 and idf(fast) = ln(4/2) = ln 2 ≈ 0.693. The word "smooth" appears only in D1, so df(smooth) = 1 and idf(smooth) = ln(4/1) = ln 4 ≈ 1.386 — exactly double the weight of "fast," because "smooth" is twice as rare and therefore twice as informative about which specific review you're looking at. In D1, both words occur once (tf = 1), so tfidf(fast, D1) = 1 × 0.693 = 0.693 while tfidf(smooth, D1) = 1 × 1.386 = 1.386. A classifier fed TF-IDF vectors instead of raw counts automatically pays more attention to "smooth" than to "fast" when representing D1 — exactly the behaviour you want, derived from a formula rather than asserted. For the Naive Bayes derivation below we deliberately use plain counts, because Naive Bayes has its own principled way of handling word frequency (through probability, not weighting) — the two techniques are not meant to be combined naively, and knowing when to use raw counts versus TF-IDF is itself part of pipeline design.

Stage 4: The Classifier — Deriving Naive Bayes

The goal at this stage is to compute P(positive | review) — the probability the review is positive, given the words it contains. Start from Bayes' theorem, which you will formalize fully in Class 12 probability but whose logic you can use correctly right now:

P(class | words) = P(words | class) × P(class) / P(words)

Since P(words) is identical for the positive and negative hypotheses (it doesn't depend on the class we're testing), we only need to compare the numerators P(words | class) × P(class) across classes and normalize afterward — we never need to compute P(words) itself. The remaining problem is P(words | class): the probability of an entire multi-word review given the class. Computing this exactly would require enormous amounts of training data (every possible word combination, seen many times). The "naive" independence assumption that gives the algorithm its name says: treat each word's probability as independent of the others, given the class. That turns one hard joint probability into a product of easy ones:

P(w₁, w₂, ..., wₘ | class) ≈ P(w₁|class) × P(w₂|class) × ... × P(wₘ|class)

This assumption is technically false — "not" and "comfortable" are clearly not independent — yet the classifier still works well in practice, because getting the direction of each word's contribution right, even with a flawed independence model, is usually enough to get the final ranking right. This gap between a model's assumptions and reality, and the fact that it still performs well, is worth remembering: it recurs constantly in machine learning.

Now estimate P(word | class) from training counts: it should be (count of word in that class's documents) divided by (total words in that class). But this breaks the moment a test review contains a word the training set never associated with a class — that word's probability is exactly 0, and since we multiply, the entire product collapses to 0 regardless of how strongly every other word points the other way. Laplace (add-one) smoothing fixes this by pretending every vocabulary word was seen one extra time in every class:

P(word | class) = ( count(word, class) + 1 ) / ( total_words(class) + |V| )

where |V| is the vocabulary size. Check that this still sums to 1 across the vocabulary, which any valid probability distribution must: summing the numerator over all |V| words gives total_words(class) + |V| (each of the |V| words contributes its raw count plus 1) — exactly the denominator. So the smoothed distribution is still a legitimate probability distribution, not an ad-hoc hack.

Now derive an actual classification by hand, using the same four-review corpus. Vocabulary: {booking, fast, smooth, app, reliable, failed, slow, crashed}, so |V| = 8. Positive class has 6 total word tokens (D1 + D2, three words each); negative also has 6. Priors: P(positive) = P(negative) = 2/4 = 0.5, since two of the four training reviews are positive.

Classify the new review "booking app fast." For the positive class: count(booking, pos) = 1, count(app, pos) = 1, count(fast, pos) = 2 (once in D1, once in D2). So:

P(booking|pos) = (1+1)/(6+8) = 2/14
P(app|pos)     = (1+1)/(6+8) = 2/14
P(fast|pos)    = (2+1)/(6+8) = 3/14
unnormalized score(pos) = 0.5 × (2/14) × (2/14) × (3/14) ∝ 0.5 × 12

For the negative class: count(booking, neg) = 1, count(app, neg) = 1, but count(fast, neg) = 0 — "fast" never appeared in a negative training review:

P(booking|neg) = (1+1)/14 = 2/14
P(app|neg)     = (1+1)/14 = 2/14
P(fast|neg)    = (0+1)/14 = 1/14
unnormalized score(neg) = 0.5 × (2/14) × (2/14) × (1/14) ∝ 0.5 × 4

The two denominators (14³) and the shared factors (2/14 for "booking" and "app" in both classes) cancel in the ratio, leaving score(pos) : score(neg) = 12 : 4 = 3 : 1. Normalizing so the two probabilities sum to 1: P(positive | "booking app fast") = 3/4 = 0.75 and P(negative | ...) = 1/4 = 0.25. The single word "fast" — the only word among the three that appeared exclusively in positive training reviews — is what tips the entire verdict to 75% positive; "booking" and "app" appeared equally in both classes and contributed no discriminating signal, exactly as their identical probabilities in both classes predict.

Here is the same computation as runnable code, using log-probabilities so that multiplying many small numbers together doesn't underflow to zero — a real numerical-stability concern once a review has 30+ words instead of 3:

import math
from collections import defaultdict

train_data = [
    ("booking fast smooth", "positive"),
    ("app fast reliable", "positive"),
    ("booking failed slow", "negative"),
    ("app crashed slow", "negative"),
]

def tokenize(text):
    return text.lower().split()

vocab = set()
word_counts = {"positive": defaultdict(int), "negative": defaultdict(int)}
class_word_totals = {"positive": 0, "negative": 0}
class_doc_counts = {"positive": 0, "negative": 0}

for text, label in train_data:
    class_doc_counts[label] += 1
    for word in tokenize(text):
        vocab.add(word)
        word_counts[label][word] += 1
        class_word_totals[label] += 1

V = len(vocab)

def word_prob(word, label):
    return (word_counts[label][word] + 1) / (class_word_totals[label] + V)

def classify(text):
    total_docs = sum(class_doc_counts.values())
    log_scores = {}
    for label in ["positive", "negative"]:
        prior = class_doc_counts[label] / total_docs
        log_score = math.log(prior)
        for word in tokenize(text):
            log_score += math.log(word_prob(word, label))
        log_scores[label] = log_score
    top = max(log_scores.values())
    exp_scores = {l: math.exp(s - top) for l, s in log_scores.items()}
    total = sum(exp_scores.values())
    return {l: round(v / total, 4) for l, v in exp_scores.items()}

print("Vocabulary size:", V)
print(classify("booking app fast"))

Trace the key parts: vocab collects all 8 distinct words, so V = 8, matching the hand derivation. class_word_totals["positive"] accumulates 3 (from D1) + 3 (from D2) = 6, and the same for negative. Inside classify, subtracting the maximum log-score (top) before exponentiating is the standard numerical-stability trick — it doesn't change the final ratio (since it's the same constant subtracted from both), only keeps the intermediate numbers from underflowing. The printed output is exactly Vocabulary size: 8 followed by {'positive': 0.75, 'negative': 0.25}, matching the by-hand computation to four decimal places.

Now feed the model the review that opens the pipeline diagram above, "app crashed again, booking failed" (after tokenizing and dropping the out-of-vocabulary word "again"). Following the identical procedure: count(app,pos)=1, count(crashed,pos)=0, count(booking,pos)=1, count(failed,pos)=0, giving unnormalized positive score ∝ 2 × 1 × 2 × 1 = 4. For negative: count(app,neg)=1, count(crashed,neg)=1, count(booking,neg)=1, count(failed,neg)=1, giving ∝ 2 × 2 × 2 × 2 = 16. Ratio 4:16 = 1:4, so P(positive) = 1/5 = 0.20 and P(negative) = 4/5 = 0.80 — the 20%/80% split shown in the diagram, correctly driven by "crashed" and "failed," both words the model has only ever seen in negative training reviews.

Stage 5: Evaluation — Did the Pipeline Actually Work?

A classifier that outputs a number is not automatically a good classifier; you must measure it against reviews it never trained on. Build a confusion matrix from 10 held-out test reviews (5 genuinely positive, 5 genuinely negative) run through the trained model:

                  Predicted: Positive   Predicted: Negative
Actual: Positive        TP = 4                FN = 1
Actual: Negative        FP = 2                TN = 3

Check the row totals first, because they must match your test set by construction: row 1 (actual positive) sums to 4 + 1 = 5, row 2 (actual negative) sums to 2 + 3 = 5 — both correct. From these four counts, three standard metrics are derived, each answering a different question:

Accuracy  = (TP + TN) / total          = (4 + 3) / 10  = 0.70
Precision = TP / (TP + FP)             = 4 / 6         ≈ 0.667
Recall    = TP / (TP + FN)             = 4 / 5         = 0.80
F1        = 2 × P × R / (P + R)  = 2(0.667)(0.8)/(0.667+0.8) ≈ 0.727

Accuracy asks "what fraction of all predictions were correct?" — but it is misleading on imbalanced data: if 90 of 100 reviews were positive, a classifier that predicts "positive" every single time scores 90% accuracy while learning nothing. Precision asks "of the reviews I flagged as positive, how many actually were?" — this matters when false alarms are costly, such as auto-approving a product for a "top rated" badge based on predicted-positive reviews. Recall asks "of the reviews that were actually positive, how many did I catch?" — this matters when missing a case is costly, such as failing to flag a genuinely negative safety complaint. F1 is the harmonic mean of the two, penalizing a large gap between precision and recall more heavily than a simple average would — a model with precision 1.0 and recall 0.1 has an arithmetic mean of 0.55 but an F1 of only about 0.18, correctly signalling that the model is nearly useless despite the flattering average.

Where the Pipeline Still Breaks

Bag-of-words Naive Bayes, even with negation handling, has a real ceiling. Sarcasm ("Wonderful, my booking failed for the third time") uses purely positive vocabulary to express negative sentiment — no amount of word-level modeling catches this without world knowledge or context the model was never given. Word order beyond simple negation scope is invisible to a bag-of-words model: "the sequel was better than the original" and "the original was better than the sequel" produce identical feature vectors. And a model trained on movie reviews carries vocabulary biases that make it perform worse on, say, restaurant reviews — a phenomenon called domain shift. These are exactly the gaps that motivate the next generation of techniques you'll meet in later NLP chapters: word embeddings (which capture that "good" and "great" are similar without needing identical vocabulary matches) and transformer models like BERT (which process entire sentences with attention to word order and context, not independent word counts). Naive Bayes is not obsolete — it is fast, interpretable (you can always see exactly which words drove a verdict, as you just did by hand), and a legitimate baseline that any fancier model must beat to justify its added complexity.

Exam Connections

The Bayes' theorem derivation in Stage 4 is not a side note — it is the exact structure of the conditional-probability and Bayes' theorem questions you will formalize in CBSE Class 12 Probability, and it appears directly in JEE Main/Advanced and BITSAT probability sections, usually phrased as urns, dice, or diagnostic-test problems using the identical "posterior proportional to likelihood times prior" logic you just used on words instead of balls. The independence assumption and the smoothing derivation (proving the distribution still sums to 1) are the kind of "prove this is well-defined" reasoning KVPY and Olympiad-style questions reward over plug-and-chug. For GATE-foundation exposure, precision/recall/F1 are the standard vocabulary of every classification evaluation question you will meet again in a full machine-learning course.

Summary

A sentiment pipeline is five stages, each with a specific job: tokenize and clean to remove noise while preserving structure words like "not" and "but"; handle negation explicitly, because unigram word-counting cannot represent "not comfortable" correctly on its own; convert words to numbers via bag-of-words or TF-IDF, where TF-IDF's idf = ln(N/df) down-weights words common across documents; classify with Naive Bayes, derived from Bayes' theorem plus a conditional-independence assumption, with Laplace smoothing preventing single unseen words from zeroing out an entire prediction; and evaluate on held-out data with accuracy, precision, recall, and F1, none of which alone tells the full story. Every number in this chapter — the 0.75/0.25 split, the 0.20/0.80 split, the 0.70 accuracy, the 0.727 F1 — was derived from a formula you can re-derive yourself, not asserted.

Test Yourself

1. Using the four-review training corpus in this chapter, hand-compute P(positive | "smooth reliable"). (Check: both words appear only in positive training reviews, so the negative score should collapse toward the smoothing floor.)
2. Why does Laplace smoothing add exactly |V| to the denominator rather than some other number? Re-derive the sum-to-1 proof in your own words.
3. A test set has 20 actual-positive and 80 actual-negative reviews. A classifier predicts "negative" for everything. Compute its accuracy, precision, and recall for the positive class, and explain why accuracy alone would mislead a reviewer of this model.
4. Rewrite handle_negation so that "however" also resets negation scope, and trace it on "the app is slow, however booking is not difficult."
5. Explain, using the sarcasm example, exactly which independence assumption Naive Bayes needs sarcasm to violate in order to fail — be precise about which two "words" (or clauses) are not actually independent.

Think About It

Think about this: How would you explain sentiment analysis pipeline: building end-to-end to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.