Type this sentence into your phone and read it once: "Can you meet me at the bank at 5pm?" You understood it instantly — probably picturing a riverbank or an SBI branch depending on where you live, and settling on the bank because "meet" and "5pm" made it obvious this was about a place, not a financial account. You did all of that without noticing. Now hand that same sentence to a computer. It doesn't see a bank, a meeting, or a time. It sees this:
C a n y o u m e e t m e a t t h e b a n k a t 5 p m ?
A row of characters — bytes in memory, nothing more. There is no meaning attached to any of them. The word "bank" is not connected to "money" or "river" inside the computer the way it is inside your brain, which has spent years hearing "bank" used in both senses and picking the right one from context without effort. This is the entire problem that a field of computer science called Natural Language Processing (NLP) exists to solve: how do you get a machine that only understands numbers to do something useful with human language, which is messy, ambiguous, and full of context that isn't written down anywhere in the sentence itself?
NLP is one of the three domains you'll meet if you look at how the CBSE AI curriculum organises the subject — alongside Computer Vision (teaching computers to interpret images) and Data Science (finding patterns in numbers). NLP is specifically about text and speech: search engines, spelling correction, spam filters, IRCTC's AskDISHA chatbot that answers ticket queries in a mix of English and Hindi, and voice assistants that transcribe what you say all rely on it. This chapter builds the pipeline that all of these systems use, step by step, with code you can trace by hand.
Why "just reading" is not simple for a computer
Before building anything, it's worth sitting with why this is hard, because the difficulty is exactly what shapes every technique that follows. Consider these two sentences:
- "I saw her duck."
- "I saw her duck under the table."
In the first sentence, "duck" is almost certainly a noun — a bird she owns. In the second, "duck" is a verb — the act of ducking down. Nothing about the word "duck" itself changed. What changed was the words around it. A human resolves this instantly using context; a computer has no built-in sense of context unless something is built to supply it. Multiply this kind of ambiguity across every sentence, every language, every accent and typo, and you can see why NLP is not a solved problem even in 2026 — it is a problem that gets chipped away at, one technique layered on another.
The practical strategy computer scientists use is to break "understanding language" into a sequence of smaller, well-defined steps, each of which turns messier input into slightly cleaner, more structured output. That sequence is usually: tokenization → normalization → vectorization → some task-specific model. We'll build each stage for real, using a genuine worked example, and trace the code so every number is verifiable rather than asserted.
Step 1: Tokenization — splitting text into pieces a program can work with
The very first thing any NLP system must do is decide what counts as one "unit" of language. These units are called tokens, and for most everyday tasks a token is a word. The process of splitting a string of text into tokens is called tokenization.
The naive approach is to split on spaces. In Python:
text = "Can you meet me at the bank at 5pm?"
tokens = text.lower().split()
print(tokens)
Tracing this by hand: text.lower() converts every character to lowercase, giving "can you meet me at the bank at 5pm?". Then .split() breaks the string wherever it finds one or more spaces. The output is:
['can', 'you', 'meet', 'me', 'at', 'the', 'bank', 'at', '5pm?']
Look closely at the last token: '5pm?'. The question mark stuck to "5pm" because .split() only knows about spaces — it has no idea that punctuation is a separate thing from the word it's attached to. This is a genuine bug, not a minor cosmetic issue: if you were later counting how often "5pm" appears across many messages, "5pm" and "5pm?" would be counted as two completely different words, silently corrupting your counts.
A more careful tokenizer uses a regular expression — a pattern that describes what a valid token looks like — instead of just splitting on spaces:
import re
text = "Can you meet me at the bank at 5pm?"
tokens = re.findall(r"[a-z0-9]+", text.lower())
print(tokens)
Here, re.findall(r"[a-z0-9]+", ...) scans the lowercased string and pulls out every maximal run of letters and digits, throwing away anything that isn't a letter or digit — including the question mark. Tracing it: the scanner walks left to right, collects "can" (stops at the space), collects "you", and so on, and when it reaches "5pm?" it collects the letters and digits "5pm" and simply stops before the ?, because ? doesn't match the pattern. The output is:
['can', 'you', 'meet', 'me', 'at', 'the', 'bank', 'at', '5pm']
This is the version real systems use. Notice that tokenization already made a decision that changes meaning: it treated "5pm" as one token. A different tokenizer might have split it into "5" and "pm". There is no single "correct" tokenizer — only ones that are better or worse suited to a particular task, which is itself an important idea: NLP is full of design choices, not one fixed correct algorithm.
Step 2: Normalization — making different forms of the same word match
Tokenization gives you a list of words, but a computer still treats "run", "running", and "ran" as three completely unrelated strings — there's no built-in link between them. If you're trying to count how often a message talks about "running" (say, to detect fitness-related tweets), missing two out of three spellings loses two-thirds of your data. Normalization is the step where you try to collapse different forms of a word down to one shared form.
One common normalization technique is stemming: chopping common suffixes off a word to get at its "stem". Here is a simplified stemmer, small enough to trace completely:
def simple_stem(word):
for suffix in ("ing", "ed", "es", "s"):
if word.endswith(suffix) and len(word) - len(suffix) > 2:
return word[: -len(suffix)]
return word
for w in ["batted", "running", "played", "matches", "stopped"]:
print(w, "->", simple_stem(w))
Trace it word by word. For "batted" (6 letters): it ends with "ed", and 6 - 2 = 4, which is greater than 2, so the function returns word[:-2], which removes the last two characters and leaves "batt". For "played" (6 letters): same suffix check, word[:-2] gives "play". For "matches" (7 letters): it ends with "es", 7 - 2 = 5 > 2, so word[:-2] gives "match". For "running" (7 letters): ends with "ing", 7 - 3 = 4 > 2, so word[:-3] gives "runn". For "stopped" (7 letters): ends with "ed", 7 - 2 = 5 > 2, so word[:-2] gives "stopp". The full output is:
batted -> batt
running -> runn
played -> play
matches -> match
stopped -> stopp
This is worth stopping on, because it corrects a common misconception. Many students assume "normalization" or "stemming" is a solved, mechanical step that always produces the correct root word. Look at the output: "played" and "matches" came out correctly ("play" and "match"), but "batted", "running", and "stopped" came out wrong ("batt", "runn", "stopp" — none of which are real words, and none of which match each other's doubled-consonant pattern consistently). The reason is that English doubles the final consonant before adding "-ing" or "-ed" to short words ("bat" → "batted", "run" → "running", "stop" → "stopped"), and a suffix-stripping rule that doesn't know this grammar rule gets it wrong. Real stemmers, like the widely used Porter Stemmer, add extra rules specifically to detect and undo consonant doubling. An even more reliable (but more expensive) alternative is lemmatization, which looks words up in a dictionary of known word forms instead of guessing from suffix patterns alone. The lesson: every stage of an NLP pipeline is an approximation, and the approximations fail in specific, traceable ways — not randomly.
Step 3: Vectorization — turning words into numbers
Tokenized, normalized text is still text. But every model underneath an NLP system — from a simple rule-based classifier to a large language model — does arithmetic, not reading. At some point, words have to become numbers. The simplest and most instructive way to do this is called the Bag of Words model.
The idea: build one master list of every distinct word that appears anywhere in your data (called the vocabulary), then represent each sentence as a list of counts — how many times each vocabulary word appears in that sentence. Order within the sentence is thrown away entirely (hence "bag", not "sentence" — you're just counting what's in the bag, not the order you put things in).
Let's build this for real, using two lines of cricket commentary:
sentence_a = "India batted brilliantly and won the match"
sentence_b = "Pakistan batted poorly and lost the match"
def tokenize(t):
return t.lower().split()
tokens_a = tokenize(sentence_a)
tokens_b = tokenize(sentence_b)
vocabulary = sorted(set(tokens_a) | set(tokens_b))
print(vocabulary)
def vectorize(tokens, vocab):
return [tokens.count(w) for w in vocab]
vector_a = vectorize(tokens_a, vocabulary)
vector_b = vectorize(tokens_b, vocabulary)
print(vector_a)
print(vector_b)
Trace it. tokens_a is ['india', 'batted', 'brilliantly', 'and', 'won', 'the', 'match'] — 7 tokens. tokens_b is ['pakistan', 'batted', 'poorly', 'and', 'lost', 'the', 'match'] — 7 tokens. set(tokens_a) | set(tokens_b) takes the union of both sets of unique words (10 distinct words total, since "batted", "and", "the", and "match" appear in both sentences and are only counted once), and sorted() puts them in alphabetical order:
['and', 'batted', 'brilliantly', 'india', 'lost', 'match', 'pakistan', 'poorly', 'the', 'won']
Now vectorize walks through this 10-word vocabulary in order and, for each word, counts how many times it appears in the sentence's token list. For sentence A, checking each vocabulary word in order — "and" appears once, "batted" once, "brilliantly" once, "india" once, "lost" zero times (it's not in sentence A at all), "match" once, "pakistan" zero times, "poorly" zero times, "the" once, "won" once — giving:
[1, 1, 1, 1, 0, 1, 0, 0, 1, 1]
For sentence B, the same walk gives "and":1, "batted":1, "brilliantly":0, "india":0, "lost":1, "match":1, "pakistan":1, "poorly":1, "the":1, "won":0:
[1, 1, 0, 0, 1, 1, 1, 1, 1, 0]
Two completely different-looking sentences about cricket have become two lists of 10 numbers — the same length, directly comparable, arithmetic-ready. This is what "understanding" means to a machine learning model: not reading meaning, but comparing patterns of numbers. Where the two vectors have a 1 in the same position, both sentences used that word; where one vector has a 1 and the other has a 0, the sentences differed. You can already see, just by eyeballing the two vectors, that they overlap on the "neutral" match-reporting words (and, batted, match, the) and differ sharply on exactly the words that carry the actual news (brilliantly/won versus poorly/lost).
Step 4: A simple sentiment score built from the vectorized words
This overlap-and-difference pattern is precisely how a basic sentiment analysis system works. Instead of doing anything as complex as reading, you keep two small lists of words: ones associated with positive sentiment, and ones associated with negative sentiment. Then you count matches:
positive_words = {"brilliantly", "won"}
negative_words = {"poorly", "lost"}
def score(tokens):
pos = sum(1 for t in tokens if t in positive_words)
neg = sum(1 for t in tokens if t in negative_words)
return pos - neg
print(score(tokens_a))
print(score(tokens_b))
Tracing sentence A's tokens against the word lists: "brilliantly" is in positive_words (pos count becomes 1), "won" is in positive_words (pos count becomes 2); none of A's tokens are in negative_words, so neg stays 0. The score is 2 - 0 = 2, a positive number. For sentence B: "poorly" and "lost" are both in negative_words, giving neg = 2, and pos = 0, so the score is 0 - 2 = -2, a negative number. Output:
2
-2
This lexicon-based approach (real production systems like VADER work on this same principle, with much larger, carefully tuned word lists and extra rules for negation and intensity words like "not" and "very") is genuinely used for quick sentiment tagging — for instance, scanning thousands of product reviews or tweets about an IPL match to get a rough sense of public mood without reading each one. It is fast and easy to explain, and it is also easy to break: it has no concept of context.
The limits: what this pipeline still cannot do
A second common misconception is worth naming directly: many students assume that once a computer can process language into a sentiment score or a chatbot reply, it "understands" the sentence the way a person does. It does not, and the pipeline above shows exactly why. Feed the scorer this sentence: "The bowling was not poorly executed at all." A human reads this as a compliment, wrapped in a double negative. But tokenize would produce a token list containing "poorly", the word-matching scorer would see "poorly" in negative_words, and it would confidently score the sentence as negative — the exact opposite of its real meaning. Nothing in Bag of Words or simple lexicon matching tracks word order or negation, so "not poorly" and "poorly" look identical to it. This is precisely the "I saw her duck" problem from the start of the chapter, and it's why modern systems (the ones behind tools like Google Translate or ChatGPT) use far more sophisticated models — ones that track the order and relationships between words, not just which words are present. Those models, built on ideas called word embeddings and attention, are built on top of exactly the tokenization and vectorization ideas in this chapter; they just replace "count how many times each word appears" with far richer number representations. That is a topic for later study — the pipeline you traced by hand today, tokenize → normalize → vectorize → apply a model, is the same skeleton every one of those advanced systems still uses underneath.
The pipeline, visualized
Where this fits on the CBSE map, and what comes next
If you've studied the "AI Project Cycle" in your Artificial Intelligence coursework, this chapter has been building the "Data Acquisition" and "Data Exploration" stages specifically for text data — collecting raw sentences and turning them into a form a model can be trained on. The four-stage pipeline here (tokenize, normalize, vectorize, apply a model) is the standard structure you'll see again whenever text is involved, whether the end task is spam detection, a chatbot, autocomplete, or machine translation. What changes between applications is mainly the last stage — what kind of model consumes the numeric vectors — not the first three, which stay largely the same.
One more thing worth remembering precisely: Bag of Words, by design, discards word order. "Dog bites man" and "Man bites dog" produce the exact same vector, since both sentences use exactly the same three words. If you needed a model to tell those two sentences apart — and for most real applications, you absolutely do — Bag of Words alone is not enough, and you'd need to move to techniques that preserve sequence, such as tracking pairs of consecutive words (called bigrams) or the sequence-aware models mentioned earlier. Knowing what a technique cannot do is as important as knowing what it can.
Check your understanding
- Using the regular-expression tokenizer
re.findall(r"[a-z0-9]+", text.lower()), what tokens would it produce from the string"AI@school, 2026!"? Trace it character by character before checking your answer. - Run
simple_stemfrom this chapter on the word"buses". Is the output a real English root word? Explain, using the same reasoning used for "batted" and "matches" above, why it succeeds or fails. - Build the Bag of Words vocabulary and vector by hand for the sentence
"India won and Pakistan lost", using the same 10-word vocabulary from Stage 3 of the diagram (and, batted, brilliantly, india, lost, match, pakistan, poorly, the, won). Write out the full 10-number vector. - Using the lexicon scorer from this chapter (
positive_words = {"brilliantly", "won"},negative_words = {"poorly", "lost"}), what score would it assign to"India won and Pakistan lost"? Does that score reflect that the sentence reports both a win and a loss in the same breath — and if not, what does that reveal about the scorer's blind spot? - Explain in your own words, using the "I saw her duck" and "not poorly executed" examples, why passing the Bag of Words / lexicon-scoring pipeline does not mean a program understands language the way a human reading the same sentence would.
Think About It
Think about this: How would you explain natural language processing: teaching computers to read to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where natural language processing: teaching computers to read is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting natural language processing: teaching computers to read to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind natural language processing: teaching computers to read, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.