Natural Language Processing: Making Machines Understand Language

Read this sentence once, quickly, and answer without overthinking it: "I saw her duck." Did you picture an animal, or did you picture a person quickly lowering their head to dodge something? Both readings are completely correct English. If a cricket ball had just been mentioned, you would have pictured the second. If a pond had just been mentioned, you would have pictured the first. Your brain silently used the surrounding situation to pick one meaning in a fraction of a second, and you probably did not even notice it was making a choice.

Now hand that exact sentence, with none of that surrounding situation, to a computer program. There is nothing in the spelling, the punctuation, or the grammar rules of English that forces one meaning over the other. Two entirely different sentences are hiding inside the same five words, and the machine has no pond, no cricket match, no lived experience to break the tie. This is the real problem that Natural Language Processing (NLP) exists to solve: teaching a computer, which fundamentally only understands numbers and logical rules, to work with human language — text and speech — in a way that captures at least some of what we actually mean, not just the letters we typed.

Two Different Ways a Sentence Can Be Ambiguous

Before we can build anything, we need to be precise about what makes language hard, because "it's ambiguous" is too vague to code against. There are two distinct kinds of ambiguity, and mixing them up is a common source of confusion.

Lexical ambiguity happens at the level of a single word: one word, more than one unrelated meaning. Consider "duck" on its own — "The duck swam across the pond" versus "Please duck under the low doorway." Same four letters, same part-of-speech slot the sentence allows it to fill, completely different meanings. A dictionary lookup alone cannot resolve this; the program needs the words around it.

Syntactic ambiguity (also called structural ambiguity) happens at the level of the whole sentence: every individual word has one clear meaning, but the words can be grouped together in more than one valid way. Consider "I saw the man with the telescope." Who is holding the telescope — did you use the telescope to see the man, or did you see a man who happened to be carrying a telescope? Every single word is unambiguous. The ambiguity comes purely from which phrase "with the telescope" attaches to.

Now look back at "I saw her duck" with fresh eyes: it is ambiguous in both ways at once. "Duck" is lexically ambiguous (bird, or the act of lowering your head), and depending on which meaning you pick, "her" attaches differently — either as a possessive modifying the noun ("her [pet] duck") or as the object of the verb "saw," with "duck" as a second verb describing what she did ("I saw her [do the act of] duck"). That is why the sentence felt like it had a little switch flipping in your head — you were resolving a lexical choice and a structural choice simultaneously, without noticing.

Why "Just Split on Spaces" Does Not Even Get You Started

NLP techniques were developed mostly around English, where words are separated by spaces, so "split the text into words" can feel like a solved, almost boring first step. India's languages break that assumption in several directions at once. Hindi and Marathi share the Devanagari script, while Tamil uses its own distinct script, and each of these languages has its own conventions for where spacing does and does not appear around punctuation, conjunct consonants, and honorifics — conventions that do not map onto English rules at all. Tamil and Telugu go further: they are agglutinative languages, meaning a single word can be built by gluing many suffixes onto a root to express what English would need an entire phrase for — somewhat the way the single English word "unbelievably" packs together "un-", "believe", and "-ably," except Dravidian languages do this far more extensively and productively. A tokenizer built only for English space-splitting will badly mis-segment text in these languages. This is not a minor footnote — building robust NLP for India's languages, most of which have far less digitized training text available than English, is still a genuinely open, active area of research, not a solved problem you can shrug off with one library function.

The Pipeline: From Raw Text to a Decision

Set the hardest problems aside for a moment and look at how a working NLP system is actually built, step by step. Almost every text-processing system, from a simple spam filter to a modern chatbot, pushes text through the same sequence of stages.

The rest of this chapter builds each of those four transformations, one at a time, with real code you can trace by hand.

Stage 1: Tokenization — Turning a String into a List of Words

A computer starts out seeing text as nothing but a single long string of characters — it has no built-in idea of where one "word" ends and the next begins. Tokenization is the process of splitting that raw string into meaningful pieces, called tokens, which are usually words or punctuation marks. Here is a simple English tokenizer, deliberately basic so we can trace it by hand:

def tokenize(text):
    text = text.lower()
    tokens = []
    word = ""
    for ch in text:
        if ch.isalpha():
            word += ch
        else:
            if word:
                tokens.append(word)
                word = ""
    if word:
        tokens.append(word)
    return tokens

print(tokenize("The movie was NOT good!"))

Trace it by hand, one character at a time. First, text.lower() turns the string into "the movie was not good!". Then the loop walks character by character: t, h, e are letters, so word becomes "the"; the space is not a letter, so the current word ("the") is appended to tokens and word resets to empty. The same thing happens for movie, was, and not, each ending at a space. Then g, o, o, d build up word = "good", and the final character, !, is not a letter, so "good" is appended too. The loop ends with word empty, so nothing more is added. The printed output is:

['the', 'movie', 'was', 'not', 'good']

Notice two design choices already baked into this tiny function: it lowercased everything (so "The" and "the" become the same token — otherwise a program would wrongly treat them as two unrelated words), and it silently dropped the exclamation mark. Real tokenizers usually keep punctuation as its own token when it carries meaning, but for now we only care about words.

Stop Words and Stemming: Two Quick Cleanups

Look at that token list again: the, was, and not carry very little topic information on their own — almost every English sentence contains "the." Extremely common, low-information words like these are called stop words, and many NLP systems remove them before further processing, to stop them from drowning out the words that actually matter, like "good." (Notice, however, that we deliberately keep "not" in our sentiment example later — throwing it away turns out to lose exactly the information we need. Which words count as "safe to remove" always depends on the task.)

A related cleanup is stemming: chopping words down to a shared root so that "play," "playing," and "played" are all treated as the same underlying word, rather than three unrelated tokens. A simple stemmer would map all three to the stem play. This matters because without it, a program comparing "I love playing cricket" and "I loved playing cricket yesterday" might see almost no overlap in raw tokens, even though the two sentences are about the same activity.

Stage 2: Turning Words into Numbers — the Bag of Words

Tokens are still just strings of letters, and every mathematical model — every piece of machine learning — works on numbers, not text. We need a bridge. The simplest and most widely taught bridge is the Bag of Words (BoW) model: build a fixed list of every distinct word across all the texts you care about (the vocabulary), and represent each piece of text as a list of counts — how many times each vocabulary word appears in it. The name "bag" is deliberate: word order is thrown away completely, as if you tipped all the words out of a sentence into a bag and could only count how many of each you have left.

def word_counts(tokens):
    counts = {}
    for t in tokens:
        counts[t] = counts.get(t, 0) + 1
    return counts

review1 = tokenize("great acting great story")
review2 = tokenize("great acting boring story")
print(word_counts(review1))
print(word_counts(review2))

Trace review1 first. tokenize("great acting great story") is already lowercase, and splitting on the spaces gives ["great", "acting", "great", "story"]. Feed that into word_counts: the dictionary starts empty; "great" is not yet a key, so counts.get("great", 0) returns 0, and it becomes 1; "acting" becomes 1; "great" appears again, and this time counts.get("great", 0) returns the 1 already stored, so it becomes 2; finally "story" becomes 1. The printed result is {'great': 2, 'acting': 1, 'story': 1}. The same trace on review2 gives {'great': 1, 'acting': 1, 'boring': 1, 'story': 1}.

To compare the two reviews mathematically, fix one shared vocabulary, in alphabetical order: acting, boring, great, story. Review 1 becomes the vector [1, 0, 2, 1] (one "acting," zero "boring," two "great," one "story"). Review 2 becomes [1, 1, 1, 1]. Every review, no matter how long, becomes a list of numbers exactly as long as the vocabulary — and now we can finally do arithmetic on language.

Stage 3: How Similar Are Two Pieces of Text? Cosine Similarity

Once text is a list of numbers, a very natural question becomes answerable: how similar are two pieces of text? Picture the vector [1, 0, 2, 1] as an arrow in 4-dimensional space, one axis per vocabulary word (impossible to draw, but the 2-dimensional case — two axes — works exactly the same way and is easy to picture: an arrow pointing partly right, partly up). Two arrows pointing in nearly the same direction represent texts that use words in nearly the same proportions, regardless of how long either text is. The standard way to measure "how close two directions are" is cosine similarity: the cosine of the angle between the two vectors. It ranges from -1 (exactly opposite directions — impossible here, since counts can't be negative) up to 1 (pointing in exactly the same direction, meaning identical word proportions), with 0 meaning the two vectors share no words at all.

import math

def cosine_similarity(v1, v2):
    dot = sum(a * b for a, b in zip(v1, v2))
    mag1 = math.sqrt(sum(a * a for a in v1))
    mag2 = math.sqrt(sum(b * b for b in v2))
    return dot / (mag1 * mag2)

v1 = [1, 0, 2, 1]   # acting, boring, great, story — Review 1
v2 = [1, 1, 1, 1]   # acting, boring, great, story — Review 2
print(cosine_similarity(v1, v2))

Trace the arithmetic exactly as the code does it. The dot product multiplies matching positions and adds them: (1×1) + (0×1) + (2×1) + (1×1) = 1 + 0 + 2 + 1 = 4. The magnitude of a vector is the square root of the sum of its squared entries — this is just the Pythagorean theorem extended beyond two dimensions. For v1: √(1² + 0² + 2² + 1²) = √(1 + 0 + 4 + 1) = √6 ≈ 2.449. For v2: √(1² + 1² + 1² + 1²) = √4 = 2.0. Dividing, 4 ÷ (2.449 × 2.0) = 4 ÷ 4.899 ≈ 0.816. The two reviews score about 0.82 out of a maximum of 1 — reasonably similar, which matches intuition: they share three out of four words, differing only in "great" versus "boring." If Review 2 had instead been "boring acting boring story," with no words in common at all with Review 1's "great," the shared-word terms in the dot product would vanish and the score would drop toward 0.

Stage 4: A Model That Makes a Decision — Sentiment Scoring

Now build something that actually decides something: a sentiment analyzer that reads a sentence and judges whether it expresses a positive or negative opinion. The simplest possible approach reuses everything above: give each word in a small lexicon a hand-picked score, and add up the scores of whichever lexicon words appear in the sentence.

lexicon = {"good": 1, "great": 2, "bad": -1, "terrible": -2}

def naive_sentiment(tokens):
    score = 0
    for t in tokens:
        score += lexicon.get(t, 0)
    return score

tokens = tokenize("The food was not good and the service was bad")
print(tokens)
print(naive_sentiment(tokens))

Trace the tokenizer first: lowercasing and splitting gives ['the', 'food', 'was', 'not', 'good', 'and', 'the', 'service', 'was', 'bad']. Now trace naive_sentiment: it walks the list adding each word's lexicon score, using 0 for any word not in the lexicon. "the," "food," "was," and "not" are not in the lexicon, so they each add 0. "good" is in the lexicon and adds 1, bringing the running score to 1. "and," the second "the," "service," and "was" all add 0. "bad" is in the lexicon and adds -1, bringing the score to 1 + (-1) = 0. The function returns 0 — a "neutral" verdict.

Common Misconception: "Counting Positive and Negative Words Is Enough"

A score of exactly 0 for that sentence should bother you. Read the sentence again: "The food was not good, and the service was bad." A human reads this as clearly, unambiguously negative — both halves of the sentence are complaints. The naive scorer got it wrong because Bag of Words throws away word order, and this scorer inherited that same blindness: it saw the word "good" sitting in the sentence and credited it as positive, completely ignoring that "not" standing right in front of it flips its meaning. This is a genuinely common misconception among people first learning NLP — that tallying up positive and negative words is basically "doing sentiment analysis." It is a reasonable first guess, and it is measurably wrong the moment negation, sarcasm, or comparison ("better than the terrible place next door") enters the sentence. Word order and context matter, and a model that ignores them will confidently produce wrong answers, not just imprecise ones.

A small fix handles negation directly: watch for the word "not," and flip the sign of the very next lexicon word you encounter.

def negation_sentiment(tokens):
    score = 0
    negate = False
    for t in tokens:
        if t == "not":
            negate = True
            continue
        if t in lexicon:
            value = lexicon[t]
            if negate:
                value = -value
            score += value
            negate = False
        else:
            negate = False
    return score

print(negation_sentiment(tokens))

Trace it on the same ten tokens. negate starts False and score starts 0. "the," "food," "was" are each not "not" and not in the lexicon, so the else branch just keeps negate at False. "not" is matched by the first if, so negate becomes True, and continue skips straight to the next token without touching score. "good" is in the lexicon with value 1; since negate is True, the value is flipped to -1 before being added, so score becomes -1, and negate resets to False. "and," the second "the," "service," and "was" all fall into the else branch, leaving negate at False. Finally "bad" is in the lexicon with value -1; negate is False, so the value is added unchanged: score becomes -1 + (-1) = -2. The function returns -2 — correctly and clearly negative. One extra rule, tracking a single boolean flag, was enough to fix a genuine error, and it illustrates the central lesson of this whole chapter: how you turn text into numbers determines what a program is even capable of getting right.

Stage 5 (Beyond Counting): Word Embeddings

Bag of Words has a deeper limitation than negation, and no clever fix-up rule can patch it: it treats "great" and "excellent" as two completely unrelated symbols, just as unrelated as "great" and "bicycle," because it never looks at meaning — only at whether the exact same spelling shows up again. A vocabulary of 20,000 English words gives Bag of Words 20,000 completely independent axes, with no notion that some of them are close in meaning and others are not.

Word embeddings fix this by representing each word as a point in a numeric space where distance and direction encode meaning: words used in similar contexts across a huge amount of text (a corpus) end up as nearby points, and certain directions in that space end up corresponding to consistent relationships — such as "add royalty" or "add femaleness." Real embeddings are learned automatically from data and live in hundreds of dimensions, far too many to draw. But the underlying arithmetic is simple enough to demonstrate with a tiny made-up example in just two dimensions, purely to build intuition — these exact numbers are not real trained values, just a toy illustration of the idea.

man   = (2, 1)
woman = (4, 3)
king  = (8, 9)

# "royal direction": subtract man from king
royal_direction = (king[0] - man[0], king[1] - man[1])
print(royal_direction)          # (6, 8)

# apply that same direction to "woman"
predicted = (woman[0] + royal_direction[0], woman[1] + royal_direction[1])
print(predicted)                # (10, 11)

Trace it: king[0] - man[0] = 8 - 2 = 6 and king[1] - man[1] = 9 - 1 = 8, so royal_direction = (6, 8). Adding that same offset to woman = (4, 3) gives (4 + 6, 3 + 8) = (10, 11). That predicted point lands very close to where a genuinely learned embedding would place the word "queen" — plain vector addition and subtraction on the numbers ends up performing analogy reasoning ("man is to king as woman is to ___") without the program ever being told a single grammar rule about royalty or gender.

Why This Is Still Genuinely Hard

Even with tokenization, vectors, similarity, and embeddings in hand, real language keeps finding new ways to resist. Sarcasm flips a sentence's literal meaning entirely — "Wow, five-hour power cut, exactly what I wanted today" scores strongly positive on any word-counting lexicon, because every individual word ("wow," "wanted") is positive, yet the actual sentiment is sharply negative; catching this reliably requires understanding tone and situation, not just vocabulary. Code-mixing, extremely common in everyday Indian text messages and social media — "yaar this movie was bakwaas but the songs were mast" mixes Hindi and English words inside a single sentence — breaks tokenizers and lexicons that were built assuming one language at a time. And for the majority of India's languages, there simply is not yet the enormous quantity of digitized, labelled text that English-language NLP has relied on to train large models, which is exactly why building high-quality, low-resource-language NLP remains an open research area rather than a solved engineering exercise.

Modern systems such as Google Translate or large language model chatbots handle these cases dramatically better than the toy scorers built in this chapter, but they are built from the same conceptual bricks scaled up enormously: text is still tokenized, still converted into numeric vectors (now via learned embeddings rather than raw counts), and still passed through a model — typically a large neural network called a transformer — that computes and predicts. Understanding tokenization, vectors, similarity, and the negation trap you just traced by hand is not a simplified toy version of "real" NLP set aside from the real thing — it is the actual foundation that every larger system, no matter how sophisticated, still stands on.

Check Your Understanding

1. Trace tokenize() from this chapter by hand, character by character, on the string "AI is FUN!!". What exact list does it return, and why does the double exclamation mark not produce an extra empty token?
2. Two more one-line reviews use the vocabulary [boring, funny, movie, slow] (alphabetical): Review A is "funny movie, funny scenes," Review B is "boring, slow movie." Write out the Bag-of-Words count vector for each review, then compute their cosine similarity by hand, showing the dot product and both magnitudes as separate steps.
3. Using the exact lexicon from the sentiment section, compute the score for "the trip was not bad" using naive_sentiment first, then using negation_sentiment. State in one sentence why the two answers disagree and which one better matches how a human would read the sentence.
4. Classify this sentence as syntactic ambiguity or lexical ambiguity, and explain your reasoning in one or two sentences: "Visiting relatives can be annoying."
5. In the toy embedding example, royal_direction was computed as king − man. If instead you were given prince = (5, 5) and boy = (1, -1), what 2D point would the same "royal direction" arithmetic predict for a word meaning "princess," if girl = (3, 1)? Show your subtraction and addition steps.
6. Explain, in your own words, why a bag-of-words vocabulary treats "excellent" and "great" as completely unrelated even though a human considers them near-synonyms — and name the one word from this chapter for the technique designed to fix exactly that problem.

Summary

• NLP is the field concerned with getting computers to work usefully with human language, which is fundamentally ambiguous — the same words can carry more than one valid meaning.
• Lexical ambiguity is one word with multiple unrelated meanings (like "duck"); syntactic ambiguity is one sequence of unambiguous words that can be grouped into a sentence in more than one valid way (like "I saw the man with the telescope"). Some sentences, like "I saw her duck," carry both at once.
• Space-based tokenization, the default assumption in English-centric NLP, does not transfer cleanly to Indian languages: Hindi and Marathi share the Devanagari script while Tamil uses its own, and agglutinative languages like Tamil and Telugu pack what English expresses as whole phrases into single words.
• A standard NLP pipeline moves text through five stages: raw text, tokenization (splitting into words), vectorization (turning words into numeric counts), a model (which computes over those numbers), and an output (a prediction or decision).
• Bag of Words represents a text as a vector of word counts against a fixed vocabulary, discarding word order entirely; stop words (very common, low-information words) are often removed, and stemming collapses related word forms to a shared root.
• Cosine similarity measures how close two vectors point in the same direction, computed as their dot product divided by the product of their magnitudes; it ranges from 0 (no shared words) toward 1 (near-identical word proportions).
• A word-counting sentiment scorer can be confidently wrong because it ignores order and negation — "not good" gets miscounted as positive unless the model explicitly tracks negation, which is a common misconception worth actively guarding against.
• Word embeddings place words as points in a learned numeric space where distance and direction encode meaning, so that vector arithmetic (like king − man + woman) can approximate real semantic relationships, something raw word counts can never do.
• Sarcasm, code-mixing between languages, and low-resource languages remain genuinely hard open problems; large modern systems handle them better by scaling up the same tokenize → vectorize → model → predict pipeline built in this chapter, not by replacing it.

Think About It

Think about this: How would you explain natural language processing: making machines understand language to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.