Text NLP

Imagine you message a delivery app's support chat: "recieved wrong itme yesterday, very upset, refund kab milega???" Within seconds, a bot replies asking if you'd like to start a return for the wrong item, and flags your message as urgent. No human read it. A program did — and that program had to deal with a spelling mistake ("recieved", "itme"), a mix of English and Hindi ("kab milega"), three question marks instead of one, and no explicit statement of what you actually want. A computer, at its core, only understands exact matches and numbers. So how did it get from your messy sentence to "this customer wants a refund and is angry"?

That gap — between human text, which is messy, ambiguous, and full of shortcuts, and a program, which needs precise, countable, comparable data — is exactly what Natural Language Processing (NLP) is built to close. NLP is the branch of computer science that turns unstructured text into structured information a program can act on: word counts, categories, scores, keywords. In this chapter, you will build a real NLP pipeline yourself, piece by piece, on real worked examples — not toy definitions — and you'll see exactly where the simplest approaches break, and how to fix them.

What Problem Is NLP Actually Solving?

Before writing any code, be precise about the problem. A computer program is really just a set of instructions operating on data it can compare and count — numbers, exact strings, list positions. Human language does not arrive in that form. The same idea can be written a dozen different ways: "not good", "bad", "disappointing", "could've been better" all mean roughly the same thing to a human, but to a program doing exact string comparison, they are four completely unrelated pieces of text.

NLP is the set of techniques that convert raw text into a form a program can compute over — usually a list of discrete units (words), then counts, categories, or scores built from those units. Every NLP system, from a basic keyword search to a modern chatbot, starts with the same first step: deciding what counts as one "unit" of text. That first step is called tokenization, and it is where we will begin.

Step 1: Tokenization — Breaking Text into Pieces

A token is one unit of text a program treats as a single item — usually a word, though it can also be a punctuation mark or a piece of a word. Tokenization is the process of splitting a string of text into a list of tokens. It sounds trivial — "just split on spaces" — but let's actually test that idea on a real sentence.

sentence = "I don't like the slow app!"
tokens = sentence.split()
print(tokens)

Tracing this line by line: sentence.split() with no argument splits on any run of whitespace and discards the whitespace itself. It does not know anything about punctuation, so it leaves every punctuation mark exactly where it was, glued to the nearest word. The output is:

['I', "don't", 'like', 'the', 'slow', 'app!']

Look closely at the last token: 'app!'. That is not the word "app" — it is a six-character string containing a letter sequence and an exclamation mark. If another sentence contains the word "app" without an exclamation mark, a program comparing these as exact strings would treat 'app!' and 'app' as two completely different tokens, even though a human reading both sentences knows they're about the same thing. This is the first real bug you'd hit in a naive NLP pipeline, and it's worth sitting with: split() is not tokenization, it's a rough first approximation of it.

A more careful approach uses a regular expression — a pattern that describes what a valid token looks like — instead of just splitting on spaces:

import re

sentence = "I don't like the slow app!"
tokens = re.findall(r"[A-Za-z']+", sentence)
print(tokens)

Here, re.findall(r"[A-Za-z']+", sentence) scans the string left to right and pulls out every maximal run of characters that are letters (A-Z or a-z) or apostrophes. Tracing it: it finds "I", skips the space, finds "don't" (the apostrophe is allowed inside the pattern so "don't" survives as one token instead of being split or losing its apostrophe), skips the space, finds "like", "the", "slow", and finally "app" — and stops there, because "!" is not in the character class, so it's simply excluded rather than glued on. The output is:

['I', "don't", 'like', 'the', 'slow', 'app']

Now "app" in this sentence and "app" in any other sentence are the exact same token, which is what lets a program count and compare them correctly. This is the core lesson of tokenization: it is a design decision about what should count as "the same word", and different choices (keep apostrophes or split them, keep numbers or not, treat hyphenated words as one token or two) produce genuinely different downstream results. There is no single "correct" tokenizer — only one that matches what your task needs.

Step 2: Normalization — Making Text Consistent

Once you have tokens, a second mismatch shows up: "Delivery", "delivery", and "DELIVERY" are, character by character, three different strings, even though they mean the same word. If you're counting how often "delivery" appears in customer messages, you don't want these treated as three separate words with a count of one each. Normalization is the step of converting tokens into a consistent form — most commonly, lowercasing everything.

tokens = ["Delivery", "delivery", "DELIVERY"]
normalized = [t.lower() for t in tokens]
print(normalized)

This list comprehension calls .lower() on each token in turn. .lower() converts every uppercase letter to its lowercase form and leaves everything else unchanged. The output is:

['delivery', 'delivery', 'delivery']

Now all three are identical strings, and any counting or comparison step downstream will correctly treat them as the same word. Normalization can go further than lowercasing — removing accents, expanding contractions ("don't" to "do not"), or reducing words to a root form — but for this chapter, lowercasing combined with the regex tokenizer from Step 1 (which already excludes punctuation) is enough to build a working pipeline.

Step 3: Removing Stopwords — Carefully

Run tokenization and normalization on a realistic sentence and you'll notice something: most of the tokens carry very little information about what the sentence is about. Consider a batch of customer messages a support team is trying to triage:

text = "My order is late. The order tracking is not working. Order status shows pending."
tokens = re.findall(r"[a-z]+", text.lower())
print(tokens)

Tracing this: text.lower() converts the whole string to lowercase first, then re.findall(r"[a-z]+", ...) pulls out every run of lowercase letters, automatically dropping the periods. Counting the words in the original sentence in order — my, order, is, late, the, order, tracking, is, not, working, order, status, shows, pending — gives exactly 14 tokens:

['my', 'order', 'is', 'late', 'the', 'order', 'tracking', 'is',
 'not', 'working', 'order', 'status', 'shows', 'pending']

Words like "my", "is", "the" are called stopwords — extremely common function words (articles, pronouns, common verbs like "is"/"was") that appear in almost every sentence regardless of topic, so they rarely help distinguish what a specific piece of text is about. A support system trying to figure out that this message is about a delayed order doesn't gain anything from also knowing it contains "is" and "the". Removing stopwords means filtering them out before counting:

stopwords = {"my", "is", "the", "a", "an", "of"}
filtered = [t for t in tokens if t not in stopwords]
print(filtered)

This keeps every token that is not in the stopwords set. Tracing through the 14 tokens: "my" is removed, "order" kept, "is" removed, "late" kept, "the" removed, "order" kept, "tracking" kept, "is" removed, "not" kept, "working" kept, "order" kept, "status" kept, "shows" kept, "pending" kept. The result:

['order', 'late', 'order', 'tracking', 'not', 'working',
 'order', 'status', 'shows', 'pending']

Notice exactly which word survives in the stopwords set above: "not" is deliberately not included, and this is not an oversight — it is the single most important design decision in this whole section. A very common mistake, made by students and by real production systems alike, is to grab a generic, ready-made stopword list off the internet and remove every word in it without checking what's inside. Most generic stopword lists include words like "not", "no", and "never" by default, because they are grammatically function words and appear very frequently. But "not" is precisely the word that flips a sentence's meaning. Strip it out, and "not working" becomes indistinguishable from "working" once you're just counting leftover words. You'll see exactly how badly this breaks a real system in the next two sections — so remember this rule: never remove a stopword without checking whether it changes meaning, especially negation words.

Step 4: Counting Words — The Bag-of-Words Model

Once text is tokenized, normalized, and (carefully) stripped of stopwords, the simplest and most widely used next step is just to count how often each remaining word appears. This is called the bag-of-words model: you represent a piece of text as a "bag" (an unordered collection with counts) of its words, throwing away the original order entirely. It sounds like it should lose too much information to be useful — and later in this chapter you'll see a case where it does — but it is still the foundation most real text-analysis systems build on, because word order is expensive to model and word frequency alone already reveals a lot.

from collections import Counter

filtered = ['order', 'late', 'order', 'tracking', 'not', 'working',
            'order', 'status', 'shows', 'pending']
freq = Counter(filtered)
print(freq.most_common(3))

Counter is a dictionary-like object built for exactly this: it counts how many times each item appears in a list. Tracing it, it builds internally: order → 3, late → 1, tracking → 1, not → 1, working → 1, status → 1, shows → 1, pending → 1 (each of these values comes directly from counting occurrences in filtered, the stopword-removed list from Step 3 — no other words exist to be counted). Its .most_common(3) method sorts these by count, highest first, and returns the top 3. Since "order" is the only word appearing more than once and every other word appears exactly once, the tie among the count-1 words is broken by the order each word was first seen in the list — "late" was seen before "tracking", which was seen before every other count-1 word — so the output is:

[('order', 3), ('late', 1), ('tracking', 1)]

Here, "order" is the clear standout with 3 occurrences out of 10 filtered tokens, while every other word ties at 1. For a support system scanning thousands of messages like this one, this single number — "order" is the dominant keyword — is exactly the kind of signal that lets a team notice "a lot of today's complaints mention orders" without a human reading every message. That is the practical payoff of the bag-of-words model: turning a pile of raw sentences into a sorted list of what people are actually talking about.

Step 5: A Simple Sentiment Score — And Why It Breaks

Word frequency tells you what a text is about. It doesn't tell you how the writer feels about it. To detect sentiment (positive vs. negative opinion) with the tools built so far, the standard beginner approach is a lexicon-based score: build a small dictionary of words tagged with a value, and add up the values of every token that matches.

positive = {"fast": 1, "good": 1, "great": 1, "happy": 1}
negative = {"slow": -1, "bad": -1, "poor": -1, "disappointed": -1}

review = "The delivery was fast but the product quality was not good. Very disappointed."
tokens = re.findall(r"[a-z]+", review.lower())
print(tokens)

Tracing the regex over the lowercased string produces 13 tokens, in this order:

['the', 'delivery', 'was', 'fast', 'but', 'the', 'product', 'quality',
 'was', 'not', 'good', 'very', 'disappointed']

Now score it the naive way — check every single token against the two dictionaries independently, with no awareness of neighboring words:

score = 0
for word in tokens:
    if word in positive:
        score += positive[word]
    elif word in negative:
        score += negative[word]
print(score)

Trace this loop token by token: "the" — not in either dictionary, skip. "delivery" — skip. "was" — skip. "fast" — in positive, add 1; running score is 1. "but", "the", "product", "quality", "was" — all skipped. "not" — not in either dictionary (it's neither a positive nor a negative word by itself), skipped. "good" — in positive, add 1; running score is 2. "very" — skipped. "disappointed" — in negative, add -1; running score is 1. The loop ends with:

1

With a simple rule of "score > 0 means Positive, score < 0 means Negative, score == 0 means Neutral", this program confidently labels the review Positive. Read the review again: "fast" delivery is the only genuinely positive thing said. The customer explicitly says quality was not good, and says they are very disappointed. Any human reading this would call it a negative review, probably a strongly negative one. The naive word-by-word scorer got it backwards, and the reason is now easy to name precisely: it scored "good" as a positive word, ignoring the "not" sitting directly in front of it. Bag-of-words counting treats every token as independent and throws away order — and the one piece of order that mattered most in this sentence was exactly the piece it threw away.

Fixing It: Let Two Tokens Talk to Each Other

The fix doesn't require abandoning the lexicon approach — it requires looking at pairs of tokens instead of single tokens, at least for negation. A pair of consecutive tokens is called a bigram (an n-gram with n = 2). The idea: whenever the word "not" is immediately followed by a positive word, treat that pair as a single negative signal instead of scoring the positive word on its own.

def sentiment_score(tokens, positive, negative):
    score = 0
    i = 0
    while i < len(tokens):
        word = tokens[i]
        if word == "not" and i + 1 < len(tokens) and tokens[i + 1] in positive:
            score -= positive[tokens[i + 1]]
            i += 2
            continue
        if word in positive:
            score += positive[word]
        elif word in negative:
            score += negative[word]
        i += 1
    return score

print(sentiment_score(tokens, positive, negative))

This uses a while loop with a manually controlled index i instead of a plain for loop, specifically so it can jump two tokens at once when it consumes a bigram. Trace it carefully against the same 13 tokens:

• i=0, "the": not "not", not in either dict. i becomes 1.
• i=1, "delivery": no match. i becomes 2.
• i=2, "was": no match. i becomes 3.
• i=3, "fast": in positive, score += 1 → score = 1. i becomes 4.
• i=4 through i=8 ("but", "the", "product", "quality", "was"): no matches. i becomes 9.
• i=9, "not": the first condition checks — is tokens[10] ("good") in positive? Yes. So instead of scoring "not" and later scoring "good" separately, it subtracts positive["good"] (which is 1) from the score in one move: score = 1 − 1 = 0. Then i += 2 skips past both "not" and "good" in one jump, so "good" never gets scored a second time on its own. i becomes 11.
• i=11, "very": no match. i becomes 12.
• i=12, "disappointed": in negative, score += (−1) → score = 0 − 1 = −1. i becomes 13.
• i=13: loop condition i < len(tokens) is now 13 < 13, which is false, so the loop ends.

The function returns −1. Under the same classification rule, that's Negative — the correct call. Nothing about the lexicon changed; the words "fast", "good", and "disappointed" still carry the exact same individual weights they had before. What changed is that the algorithm now checks whether "not" precedes a positive word before scoring that word, instead of scoring every token in isolation. This is the general lesson behind n-grams: single words (unigrams) are fast and simple to count, but they cannot represent relationships between neighboring words — negation, intensifiers ("very good" vs. "good"), or sarcasm-adjacent phrasing ("not exactly great"). Bigrams recover a small, specific slice of that lost context at the cost of a slightly more careful algorithm. Real systems often go further, checking trigrams or using statistical models, but the underlying trade-off — order costs more to compute but captures more meaning — is exactly what you just watched play out in thirteen tokens.

Where This Leaves Modern NLP

Every step you just built by hand — tokenize, normalize, count, watch bag-of-words fail on negation, patch it with bigrams — is still the literal first stage of even the most advanced language systems in use in 2026, including the large language models behind modern chatbots and translation tools. They still begin by tokenizing text (their tokenizers are more sophisticated, often splitting words into smaller sub-word pieces rather than whole words, which helps them handle spelling variants and words they've never seen before). What differs at the higher layers is how they handle exactly the problem you just solved with a hand-written bigram check: instead of a programmer writing an explicit rule for "not + positive word", these systems are trained on enormous amounts of text to automatically learn how much weight to give every word based on every other word around it — a mechanism generally called attention. But that machinery exists to solve the same core problem this chapter has been about from the first sentence: language carries meaning in the order and relationships between words, not just in which words are present, and any system that ignores order — as plain bag-of-words does — will get some sentences backwards, exactly as you just proved with real code and a real number.

Check Your Understanding

• Tokenize the sentence "ISRO's Chandrayaan mission wasn't delayed!" using the regex pattern r"[A-Za-z']+". Write out the exact list of tokens the regex would produce, in order.
• A stopword list you found online includes "no", "not", and "never". Explain, using the sentiment-scoring example from this chapter, exactly what would go wrong if you used that list unmodified on product reviews.
• Given the sentence "The food was not bad", trace the bigram-aware sentiment_score function from this chapter, assuming "bad" is in the negative dictionary with value −1 and there is no rule for "not" + negative word (only "not" + positive word is handled). What score does the function return, and does it match what a human would judge the sentence to mean? What does this tell you about the limits of the fix built in this chapter?
• A support team has 500 customer messages. Explain, step by step, how you would use tokenization, normalization, stopword removal, and frequency counting (in that order) to find the three most common complaint topics without reading all 500 messages yourself.

Summary

Text NLP begins by solving a concrete mismatch: programs compute over exact, countable data, while human language is messy, redundant, and order-dependent. Tokenization decides what counts as one unit of text — and the choice of tokenizer (plain split() versus a regex pattern) measurably changes the result, as seen when 'app!' and 'app' were treated as different tokens. Normalization (lowercasing) makes matching case-insensitive. Stopword removal focuses counting on meaningful words, but only when applied carefully — blindly removing negation words like "not" destroys exactly the information that determines a sentence's meaning, as the sentiment example proved with a wrong "Positive" label on a clearly negative review. The bag-of-words model — counting word frequency while discarding order — is powerful for topic detection (spotting that "order" dominated a batch of support messages) but fundamentally blind to relationships between words. Bigrams, which look at pairs of adjacent tokens, recover a specific, useful slice of that lost order — enough to correctly flip a "not good" review from a wrongly-positive score of +1 to a correctly-negative score of −1 — without requiring a complete redesign of the approach. Every one of these five steps remains the foundation underneath far more advanced NLP systems, including the language models used in modern chatbots and translators.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where text nlp is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting text nlp to at least 3 other topics you have studied.