Sentiment Analysis: Understanding Opinions

Two Reviews, One Question

Look at these two reviews left on a food-delivery app after the same restaurant order:

Review A: "The biryani was amazing, packed hot, and arrived in twenty minutes. Loved it!"
Review B: "The biryani was cold, the portion was tiny, and the delivery took ninety minutes. Never ordering again."

You read these in about three seconds and instantly know Review A is happy and Review B is angry. Nobody taught you a formula for this — you just understood the feeling behind the words. Now here is the actual engineering problem: Swiggy, Zomato, Flipkart and Amazon India receive millions of reviews every single day. No team of humans can read all of them to figure out which restaurants are slipping, which products are disappointing customers, or which delivery routes are too slow. A computer has to read the reviews instead. But a computer does not "feel" anything — it only sees a string of characters. So the real question this chapter answers is: how do you turn a sentence into a number that captures whether it sounds happy or angry, using nothing but arithmetic? That process is called sentiment analysis, and by the end of this chapter you will have built (and broken, and fixed) a working sentiment classifier by hand.

Give the Computer a Dictionary of Feelings

Here is the simplest idea anyone has ever had for this problem, and it turns out to work surprisingly well as a first attempt: build a dictionary where every word is tagged with a small number saying how positive or negative it usually sounds. This dictionary is called a sentiment lexicon. A tiny example lexicon might look like this:

amazing  : +2
great    : +2
good     : +1
fresh    : +1
fast     : +1
slow     : -1
cold     : -1
bad      : -2
terrible : -3
worst    : -3

Positive words get positive numbers, negative words get negative numbers, and the size of the number reflects how strong the feeling is — "terrible" (-3) is worse than "slow" (-1). Every word that is not in the dictionary — "the," "was," "biryani," "delivery" — is treated as emotionally neutral and scores 0. This is exactly the kind of dictionary a linguist or a data-labelling team builds by hand, and several such lexicons (VADER and AFINN are two well-known English examples) are freely available and used as a starting point in real NLP pipelines.

Worked Example: Scoring a Real Review

Let's score Review A by hand, step by step, exactly the way a program would.

Step 1 — Tokenize. Break the sentence into individual words (called tokens), converting everything to lowercase and dropping punctuation, since "Amazing" and "amazing," should be treated as the same word:

"The biryani was amazing, packed hot, and arrived in twenty minutes. Loved it!"

tokens = [the, biryani, was, amazing, packed, hot, and,
          arrived, in, twenty, minutes, loved, it]

Step 2 — Look up every token in the lexicon. Most words are not in our small dictionary, so they score 0. Only "amazing" is present, scoring +2. (A real lexicon would also know "loved," but ours is deliberately tiny for this example — we will use a bigger one in code shortly.)

Step 3 — Sum the scores. Total = 2. Since the total is greater than 0, we classify the review as Positive.

That single rule — add up the score of every word, and check whether the total is above, below, or equal to zero — is the entire engine of a basic sentiment analyzer. It feels almost too simple to work, yet it is genuinely how many early production systems worked, and it still powers quick sentiment tags in many dashboards today because it is fast and needs no training data.

Formalizing the Idea

Now let's write down precisely what we just did, because this exact pattern reappears constantly in AI. Sentiment analysis is the task of automatically determining the emotional tone (usually positive, negative, or neutral) expressed in a piece of text. The output is typically reported on a polarity scale — a number line where negative values mean negative sentiment, positive values mean positive sentiment, and values near zero mean neutral or mixed. Formally, for a piece of text made of tokens w₁, w₂, ..., wₙ, the naive lexicon-based sentiment score is:

score(text) = lex(w1) + lex(w2) + ... + lex(wn)

where lex(w) looks up word w in the sentiment lexicon and returns 0 if the word is not found. The classification rule is then a simple threshold:

if score > 0:  Positive
if score < 0:  Negative
if score == 0: Neutral

Let's turn this into actual code so we can run it on many reviews, not just do it by hand.

lexicon = {
    "amazing": 2, "great": 2, "good": 1, "fresh": 1, "fast": 1,
    "loved": 2, "excellent": 2,
    "slow": -1, "cold": -1, "bad": -2, "terrible": -3,
    "worst": -3, "tiny": -1
}

def naive_sentiment(text):
    words = text.lower().replace(",", "").replace(".", "").replace("!", "").split()
    score = 0
    for w in words:
        score += lexicon.get(w, 0)
    if score > 0:
        return "Positive", score
    elif score < 0:
        return "Negative", score
    else:
        return "Neutral", score

print(naive_sentiment("The biryani was amazing, packed hot, and arrived in twenty minutes. Loved it!"))
# ('Positive', 4)

Trace it: the cleaned tokens are the biryani was amazing packed hot and arrived in twenty minutes loved it. Only "amazing" (+2) and "loved" (+2) are in the lexicon, so score = 2 + 2 = 4, and since 4 > 0 the function returns ('Positive', 4). Now try Review B: naive_sentiment("The biryani was cold, the portion was tiny, and the delivery took ninety minutes. Never ordering again."). Tokens include "cold" (-1) and "tiny" (-1); nothing else matches, so score = -1 + -1 = -2, and the function correctly returns ('Negative', -2). Two lines of dictionary lookup and a loop — and it correctly separated a happy customer from an angry one.

Where the Simple Method Breaks

A very common misconception among students first meeting sentiment analysis is: "if I just build a big enough lexicon with enough words, summing scores will always work." This is false, and the cleanest way to see why is with one short sentence: "The food was not good."

Run it through naive_sentiment. The cleaned tokens are the food was not good. "Not" is not in our lexicon (it carries no sentiment of its own — it is a negator, a word whose job is to flip the meaning of what follows it), so it scores 0. "Good" scores +1. Total score = 1, so the function confidently reports Positive — even though any human reading "The food was not good" instantly hears a complaint. The word-counting approach has no concept of grammar or word order; it treats a sentence as an unordered bag of words (this is literally called the bag-of-words assumption), so "not good" contributes exactly the same score as "good" appearing alone. This single failure mode — negation — is one of the most cited limitations of naive lexicon-based sentiment analysis, and it is worth fixing by hand once so you understand exactly what "handling negation" means at the code level.

Fixing It: A Negation-Aware Scorer

The fix is a rule: whenever a negation word ("not," "no," "never," and similar) appears, flip the sign of the very next sentiment-bearing word that follows it, then stop flipping.

negation_words = {"not", "no", "never"}

def negation_aware_sentiment(text):
    words = text.lower().replace(",", "").replace(".", "").split()
    score = 0
    negate = False
    for w in words:
        if w in negation_words:
            negate = True
            continue
        word_score = lexicon.get(w, 0)
        if word_score != 0:
            if negate:
                word_score = -word_score
                negate = False   # the negation has done its job; reset it
            score += word_score
    return score

print(negation_aware_sentiment("The food was not good"))   # -1
print(negation_aware_sentiment("The biryani was amazing"))  # 2 (unaffected)

Trace the first call carefully: tokens are the food was not good. "the," "food," "was" are not negation words and score 0. "not" sets negate = True and is skipped with continue. "good" has word_score = 1, which is nonzero, and negate is currently True, so it flips to -1, adds -1 to the running total, and resets negate back to False. Final score = -1 — now correctly negative. The second call has no negation word at all, so "amazing" passes through unflipped and the score stays +2, exactly as before. Notice the design choice: negate only resets when it actually flips a nonzero scoring word, so a stray function word like "the" sitting between "not" and "good" would not accidentally cancel the negation. This is still a simplified rule — real NLP libraries use a fixed window (commonly the next 2–4 words) rather than "the next scoring word, however far away," because negation naturally fades in real sentences — but it captures the core mechanism correctly.

A Second Misconception: Words Are Not Intent

Fixing negation does not make sentiment analysis perfect, and it is important to know exactly where it still fails, rather than trusting it blindly. Consider a customer who tweets: "Wow, only three hours for a five-minute walk delivery. Fantastic service." Every sentiment word here — "wow," "fantastic" — is positive, and there is no negation word to trigger our fix. A lexicon-based scorer will report a strongly positive score. But any human recognizes this as sarcasm: the customer is furious, and using cheerful words to mock a bad experience. Sentiment analysis systems, at their core, measure the emotional charge of the words used, not the writer's true intent — and those two things usually agree but can be deliberately mismatched. This is why companies that rely on automated sentiment scores at scale (rating dashboards, brand-monitoring tools) still keep a human review process for flagged or ambiguous cases, and why more advanced models try to use surrounding context, punctuation patterns (excessive exclamation marks, quotation marks around positive words), and even emoji to catch sarcasm — a genuinely unsolved, active research problem in NLP, not a minor edge case.

A Different Idea: Learning Sentiment from Data

Everything so far required a human to sit down and manually decide that "amazing" is worth +2. That does not scale to slang, new products, or other languages, and it means the system is only as good as the dictionary someone wrote. The alternative approach — the one that powers most production sentiment systems today — is to learn which words signal positive or negative sentiment directly from a pile of already-labelled examples, instead of hand-writing the rules.

Here is the core idea with real arithmetic, using a tiny training set of eight one-line Flipkart-style phone reviews that a human has already labelled Positive or Negative:

Positive reviews (4):
  1. "excellent camera quality"
  2. "battery life is excellent"
  3. "great value for money"
  4. "excellent build quality"

Negative reviews (4):
  1. "battery drains fast"
  2. "camera is disappointing"
  3. "poor build quality"
  4. "screen cracked in a week"

Instead of deciding sentiment scores by intuition, we count. How many of the 4 positive reviews contain the word "excellent"? Three of them (reviews 1, 2, and 4) — that's 3/4 = 75%. How many of the 4 negative reviews contain "excellent"? Zero — 0/4 = 0%. That is an enormous gap, and it tells us, purely from counting, that seeing the word "excellent" is very strong evidence a review is positive — we never had to type in a number by hand; the data told us.

Compare that to the word "quality": it appears in 2 of 4 positive reviews (2/4 = 50%) and 1 of 4 negative reviews (1/4 = 25%) — it still leans positive, but far more weakly, since it shows up on both sides. And "battery" appears in exactly 1 of 4 positive reviews and 1 of 4 negative reviews (25% each) — an even split means the word carries almost no useful information for telling the two classes apart, even though it sounds like it should matter.

Now classify a brand-new, never-seen review: "excellent battery quality." "Excellent" strongly favours Positive (75% vs 0%). "Quality" mildly favours Positive (50% vs 25%). "Battery" is a tie and contributes nothing either way. Combining the evidence, the review leans clearly Positive — and we reached that conclusion without ever hand-assigning a single sentiment score; we only counted how often words showed up in reviews a human had already labelled. This idea — comparing how often a word occurs in each category and combining that evidence using probability — is the intuition behind a real, widely-used algorithm called a Naive Bayes classifier, which you will meet formally with its full probability formula in later AI coursework. The word "naive" in its name refers to the simplifying assumption that each word contributes its evidence independently of the others — an approximation, but one that works remarkably well in practice.

The trade-off between the two approaches is worth stating plainly. A hand-built lexicon needs no training data and is instantly understandable — you can always point to exactly which word caused a score — but it must be manually maintained and cannot learn from new patterns on its own. A data-driven classifier can pick up sentiment cues no dictionary-writer thought of (brand names, slang like "paisa vasool," even emoji use), and improves as you feed it more labelled reviews, but it needs that labelled training data to begin with, and its decisions are harder to explain word-by-word.

The Pipeline, Visually

Putting the lexicon-based pipeline together — tokenize, look up, apply the negation rule, sum, threshold — looks like this:

Follow the diagram with our example: "The food was not good" ends up with total = -1, which routes down the left branch to Negative — exactly matching what a human reader feels, and exactly what our fixed code returned.

Where This Fits and Where You'll See It

In the CBSE AI curriculum, sentiment analysis sits inside Natural Language Processing (NLP), and the process you just walked through — deciding what to build, gathering example text, cleaning it, scoring or modelling it, and checking whether the results make sense — is a direct instance of the AI Project Cycle (Problem Scoping → Data Acquisition → Data Exploration → Modelling → Evaluation) that frames every AI project in the syllabus. You have already done Problem Scoping (classify a review's tone), Data Acquisition (the review text), Data Exploration and Modelling (the lexicon and scoring rule), and Evaluation (checking the negation example against what a human would say).

You will meet this exact pattern across Indian consumer platforms. Swiggy and Zomato roll up sentiment scores from millions of order ratings and free-text comments into restaurant-level "quality" signals that influence search ranking. Flipkart and Amazon India use review sentiment, combined with star ratings, to flag sellers whose product descriptions do not match what customers actually receive. IRCTC and other government service apps analyze feedback-form comments to spot recurring complaints (delayed trains, unclean coaches) without needing a person to read every single submission. In every one of these, the underlying question is the one this chapter answered: turn free-form human text into a number a machine can act on, understand exactly what that number does and does not capture, and know precisely where — like negation and sarcasm — it can be fooled.

Check Your Understanding

1. Using the lexicon in this chapter, hand-trace naive_sentiment on the sentence "The service was fast and the biryani was fresh." List every token, mark which ones are found in the lexicon, and state the final score and classification.
2. Explain, in your own words, exactly why naive_sentiment("The food was not good") returns Positive while negation_aware_sentiment on the same sentence returns Negative. Refer to the bag-of-words assumption in your answer.
3. Trace negation_aware_sentiment("The delivery was never late") step by step and give its final score. (Careful: "late" is not in our lexicon at all — what does that mean for what gets flipped?)
4. In the Naive-Bayes-style example, "battery" appeared in 25% of positive reviews and 25% of negative reviews. Explain why a word with equal frequency in both classes is nearly useless for classification, even though it is a perfectly meaningful word.
5. Design a short sarcastic sentence (in English or Hinglish) about slow internet speed that uses only positive-sounding words, and explain exactly why a lexicon-based scorer would misclassify it.

Summary

Sentiment analysis converts free text into a polarity judgment — positive, negative, or neutral — so machines can process opinions at a scale no human team could match. The simplest working method is lexicon-based: tokenize the text, look up each token's score in a hand-built dictionary, sum the scores, and threshold the total against zero. This bag-of-words approach is fast and transparent but has a specific, well-understood blind spot: it ignores word order, so negation words like "not" silently break it unless you add an explicit rule to flip the sign of the word that follows a negator. Even a negation-aware scorer still cannot detect sarcasm, because it reads the emotional charge of words, not the writer's actual intent. The alternative, data-driven approach — exemplified by the Naive Bayes intuition — learns which words are strong signals of each sentiment by counting how often they appear in already-labelled positive and negative examples, trading hand-built rules for the ability to learn patterns directly from data. Both approaches remain in active use today, in exactly the kind of Indian consumer platforms — food delivery, e-commerce, rail feedback — that generate the review text this chapter used as its running example.

Think About It

Think about this: How would you explain sentiment analysis: understanding opinions to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.