POS Tagging: Understanding Grammar

The Sentence a Dictionary Cannot Read

Say this sentence out loud: "I read books every day, but yesterday I read a strange one." The word read appears twice, spelled identically, but your mouth pronounces it two different ways — "reed" the first time, "red" the second. Nobody taught you a rule that says "pronounce it 'red' when the sentence is about the past." You just knew, instantly, because the grammatical role of the word changed: the first read is present tense, the second is past tense. A text-to-speech engine reading this sentence aloud — the kind that powers screen readers, IVR systems on your bank's helpline, or a voice assistant reading out a WhatsApp message — has no ears and no intuition. It sees two identical strings of four letters and must somehow produce two different sounds. It can only do this if, before speaking, it has already worked out the grammatical category of each word in context. That silent, prior step is called part-of-speech tagging, and it is the subject of this chapter.

Here is a second, classic example that linguists have used for decades to make the same point sharper: "I saw her duck." Read it again. Did you picture a woman ducking her head to avoid something, or a pet bird belonging to her? Both readings are grammatically legal. In the first, her is a possessive pronoun and duck is a noun (an animal). In the second, her is an object pronoun and duck is a verb (an action — she lowered her head). The words on the page never change. What changes is which grammatical category, or part of speech, each word is playing. A human resolves this in milliseconds using context and, if spoken aloud, tone of voice. A machine has to compute it — and computing it correctly is the foundation on which almost every other language-processing task is built: grammar checkers, machine translation, search engines, chatbots, and speech synthesis all fail silently if this one step goes wrong.

What Part-of-Speech Tagging Actually Is

Formally, part-of-speech (POS) tagging is the task of assigning to every token in a sentence a label describing its grammatical category — noun, verb, adjective, and so on — based on both the word itself and the words around it. The output is a sequence of (word, tag) pairs, one for every token in the input:

Rahul   -> NOUN (proper noun)
bowls   -> VERB
fast    -> ADVERB

Notice the phrasing: "based on both the word itself and the words around it." That second half is the entire difficulty of the task. If every word had exactly one possible tag, POS tagging would be a one-line dictionary lookup and this chapter would not need to exist. It does need to exist because English — and Hindi, and every other natural language — is riddled with words that legitimately belong to more than one grammatical category, and only the sentence around them decides which one applies on a given occasion.

Tagsets: Choosing How Fine-Grained "Category" Should Be

Before building any tagger, you must fix a tagset — the closed list of labels you are allowed to assign. Two tagsets dominate real NLP work, and they trade off simplicity against precision.

• Universal POS tags (used by the Universal Dependencies project): a compact set of about 17 coarse categories that is designed to work across languages, including Hindi, Tamil, and Marathi, not just English. The core ones are NOUN, PROPN (proper noun), VERB, AUX (auxiliary/helping verb), ADJ, ADV, PRON, DET (determiner), ADP (adposition — prepositions in English, postpositions in Hindi), NUM, CONJ/SCONJ, PART (particle), INTJ, and PUNCT.
• The Penn Treebank tagset: a finer-grained, English-specific set of 36 word-level tags used by almost every classic English NLP tool, including Python's NLTK. It splits what Universal POS lumps into one category. Verbs alone are split into six tags depending on form: VB (base form, "bowl"), VBZ (third-person singular present, "bowls"), VBD (past tense, "bowled"), VBG (gerund/present participle, "bowling"), VBN (past participle, "bowled" as in "has bowled"), and VBP (non-third-person present, "bowl" as in "I bowl"). Nouns split into NN (singular common), NNS (plural common), NNP (singular proper), and NNPS (plural proper). Adjectives are JJ, adverbs RB, determiners DT, prepositions IN, coordinating conjunctions CC, cardinal numbers CD, and personal pronouns PRP.

The finer tagset matters in practice: knowing a word is a VERB is useful, but knowing it is specifically VBD (past tense) versus VBP (present tense) is what let you correctly pronounce "read" in the opening example, and it is what a grammar checker needs to catch a subject-verb agreement error like "he bowl fast" instead of "he bowls fast."

The Core Difficulty: The Same Word, Different Roles

Think about how many everyday English words carry more than one part of speech depending on the sentence. Book is a noun in "pass me that book" and a verb in "I will book two tickets on IRCTC." Close is a verb in "please close the door" and an adjective in "the station is close to my house." Fast is an adjective in "that was a fast ball" and an adverb in "Bumrah bowls fast." Well is an adverb in "she batted well" and a noun in "they dug a well for water." None of these are rare or exotic words — they are among the most ordinary words in the language, and that is not a coincidence. Short, high-frequency words accumulate multiple grammatical uses over centuries precisely because they get reused so often. This means that even though the majority of distinct words in a dictionary have only one possible tag, a large share of the words you actually meet in running text belong to this small, heavily overloaded, ambiguous set — so a tagger that ignores context will get a disproportionate number of real sentences wrong, even if it looks accurate on a word-by-word count.

A Misconception Worth Correcting Directly

Many students, on first meeting this topic, assume POS tagging works the way grammar was taught in school: memorize that "run" is a verb, "book" is a noun, "fast" is an adjective, and then apply the memorized label whenever the word appears. This is exactly wrong, and it is worth stating precisely why. A word does not carry a fixed part of speech the way an atom carries a fixed atomic number. It carries a set of possible parts of speech, and the sentence context selects one member of that set on each specific occasion. "Fast" is not "an adjective that is sometimes used as an adverb" — it is a word that is genuinely, symmetrically, either, and no dictionary entry alone can tell you which, for a given sentence, without looking at the neighboring words. Any tagging system — rule-based, statistical, or neural — has to be built around this fact, not around the fiction of one-word-one-tag.

From Rules to Probability

The earliest POS taggers, built in the 1960s–80s, were hand-written rule systems: "if a word ends in -ing and follows a form of 'to be', tag it VBG"; "if a word follows 'the' and is not itself a determiner, tag it a noun." Rule-based tagging is intuitive and transparent, but it does not scale — language has thousands of such regularities and just as many exceptions, and every new rule risks breaking three old ones. Modern taggers instead learn the regularities statistically from a large collection of text that humans have already tagged by hand, called an annotated corpus. The classical statistical approach — still the cleanest one to learn the mathematics from — is the Hidden Markov Model (HMM), and it is built entirely out of conditional probability and Bayes' theorem, the same tool you use in JEE/BITSAT probability problems, just applied to sequences of words instead of drawing balls from urns.

The Hidden Markov Model, Derived Step by Step

Let a sentence be a sequence of words W = w₁, w₂, …, w_n, and let T = t₁, t₂, …, t_n be a candidate sequence of tags, one per word. The tags are called "hidden" because you observe the words, but the true grammatical category behind each word is not written on the page — it must be inferred. The goal of tagging is to find the tag sequence that is most probable given the words actually observed:

T* = argmax over all T of  P(T | W)

Directly estimating P(T | W) from data is hard, because the number of possible tag sequences grows exponentially with sentence length. So we apply Bayes' theorem to flip the conditioning:

P(T | W) = P(W | T) * P(T) / P(W)

Since W is fixed once we're given a sentence to tag, P(W) is a constant that does not affect which T maximizes the expression, so we can drop it:

T* = argmax over T of  P(W | T) * P(T)

This is still not computable as written, because P(W | T) and P(T) are each probabilities over an entire sequence, and we would need to have seen every possible sentence during training to estimate them directly. The HMM makes the problem tractable with two simplifying assumptions, each of which is a genuine approximation, not a fact about language:

• Emission (output) independence: each word depends only on its own tag, not on any other word or tag in the sentence — P(W | T) ≈ P(w₁|t₁) · P(w₂|t₂) · … · P(w_n|t_n).
• Markov (bigram) independence: each tag depends only on the single tag immediately before it, not on the whole tagging history — P(T) ≈ P(t₁|t₀) · P(t₂|t₁) · … · P(t_n|t_n-1), where t₀ is a special start symbol.

Putting the two together gives the working formula every bigram HMM tagger optimizes:

T* = argmax over T of  the product, for i = 1 to n, of  P(w_i | t_i) * P(t_i | t_(i-1))

The two ingredients have names: P(w_i | t_i) is an emission probability — how likely is this specific word, given this tag? — and P(t_i | t_i-1) is a transition probability — how likely is this tag to follow the previous one? Both are estimated by simple counting on a tagged training corpus: divide how often something happened by how often it could have happened. This is exactly the "favourable outcomes over total outcomes" logic of classical probability, applied to a corpus instead of a sample space of coins or cards.

A Worked Example You Can Check by Hand

To make this concrete, build a tiny toy training corpus — six short, already-tagged sentences, using a five-tag mini-tagset {NN, VB, RB, JJ, DT} — and use it to tag a new sentence that was never seen during training.

Training corpus (hand-tagged):
1. Sachin/NN bowls/VB fast/RB
2. Rahul/NN plays/VB well/RB
3. The/DT bowls/NN are/VB clean/JJ
4. The/DT ball/NN is/VB fast/JJ
5. Rahul/NN bowls/VB well/RB
6. The/DT pitch/NN is/VB fast/JJ

Notice that "Rahul bowls fast" — the sentence we are about to tag — never appears verbatim anywhere in this corpus, so the model genuinely has to generalize from separate pieces of evidence, not recall a memorized sentence.

Counting transitions (with <s> marking the start of each sentence) and dividing by the total number of times each source tag occurred gives:

P(NN|<s>) = 3/6 = 0.5      P(DT|<s>) = 3/6 = 0.5
P(VB|NN)  = 6/6 = 1.0      P(NN|DT)  = 3/3 = 1.0
P(RB|VB)  = 3/6 = 0.5      P(JJ|VB)  = 3/6 = 0.5

Counting emissions the same way (occurrences of a specific word under a specific tag, divided by total occurrences of that tag):

P(Rahul|NN) = 2/6 = 0.333    P(bowls|VB) = 2/6 = 0.333
P(fast|RB)  = 1/3 = 0.333    P(fast|JJ)  = 2/3 = 0.667

Now tag the unseen sentence "Rahul bowls fast" using the Viterbi algorithm — the standard dynamic-programming method for finding the single most probable tag sequence without literally enumerating every combination. At each word, Viterbi keeps only the best-scoring path into each candidate tag, because any path that wasn't locally best can never become part of the globally best sequence (this pruning is what makes the algorithm run in time proportional to sentence length times tagset size, instead of exponential time).

Position 1, "Rahul": the only tag ever observed for "Rahul" is NN. Score = P(NN|<s>) × P(Rahul|NN) = 0.5 × 0.333 = 0.1667.

Position 2, "bowls": the only candidate tag for "bowls" here is VB. Score = (best path into NN) × P(VB|NN) × P(bowls|VB) = 0.1667 × 1.0 × 0.333 = 0.0556.

Position 3, "fast": now there are two live candidates, RB and JJ, and this is the genuinely ambiguous step.

Score(RB) = 0.0556 * P(RB|VB) * P(fast|RB) = 0.0556 * 0.5 * 0.333 = 0.00926
Score(JJ) = 0.0556 * P(JJ|VB) * P(fast|JJ) = 0.0556 * 0.5 * 0.667 = 0.01852

JJ scores exactly double RB, so the HMM commits to the path Rahul/NN bowls/VB fast/JJ. Read that result carefully before moving on, because it is the single most important lesson in this section: a fluent English speaker knows "Rahul bowls fast" means fastly — the adverb sense, describing the pace of the bowling, exactly like "Rahul plays well." A human would tag it RB. The statistical model tagged it JJ, and it did so for a completely defensible mathematical reason — in this toy corpus, "fast" simply occurred as an adjective (describing balls and pitches) twice as often as it occurred as an adverb. The model is not broken; it is doing exactly what it was built to do, which is maximize probability under the data it was actually shown. This is the central limitation of every statistical NLP model, HMMs included: it cannot be more correct than the data it was trained on. A production POS tagger is trained on corpora with hundreds of thousands of tagged sentences precisely to dilute this kind of accidental skew — with six sentences, one imbalanced example is enough to flip the answer.

Watching the Algorithm Run

The Python below implements exactly the computation just done by hand, using the same numbers, so every printed value can be checked against the arithmetic above.

trans = {
    ('<s>', 'NN'): 0.5, ('<s>', 'DT'): 0.5,
    ('NN', 'VB'): 1.0,
    ('DT', 'NN'): 1.0,
    ('VB', 'RB'): 0.5, ('VB', 'JJ'): 0.5,
}
emit = {
    ('NN', 'Rahul'): 1/3,
    ('VB', 'bowls'): 1/3,
    ('RB', 'fast'): 1/3,
    ('JJ', 'fast'): 2/3,
}
sentence = ['Rahul', 'bowls', 'fast']
candidates = {'Rahul': ['NN'], 'bowls': ['VB'], 'fast': ['RB', 'JJ']}

V, back = [{}], [{}]
for tag in candidates[sentence[0]]:
    V[0][tag] = trans[('<s>', tag)] * emit[(tag, sentence[0])]
    back[0][tag] = '<s>'

for i in range(1, len(sentence)):
    V.append({}); back.append({})
    word = sentence[i]
    for tag in candidates[word]:
        best_prev, best_score = None, -1
        for pt in V[i - 1]:
            score = V[i - 1][pt] * trans.get((pt, tag), 0)
            if score > best_score:
                best_score, best_prev = score, pt
        V[i][tag] = best_score * emit[(tag, word)]
        back[i][tag] = best_prev

last_tag = max(V[-1], key=V[-1].get)
path = [last_tag]
for i in range(len(sentence) - 1, 0, -1):
    last_tag = back[i][last_tag]
    path.insert(0, last_tag)

print(list(zip(sentence, path)))

Tracing it: V[0]['NN'] = 0.5 × 0.333… = 0.1667, matching Position 1 above. V[1]['VB'] = 0.1667 × 1.0 × 0.333… = 0.0556, matching Position 2. For "fast", the inner loop computes V[2]['RB'] = 0.0556 × 0.5 × 0.333 = 0.00926 and V[2]['JJ'] = 0.0556 × 0.5 × 0.667 = 0.01852 — identical to the two Score lines above. max(V[-1], key=V[-1].get) picks 'JJ' since 0.01852 > 0.00926, and the backtrace through back reconstructs NN -> VB -> JJ. The program prints:

[('Rahul', 'NN'), ('bowls', 'VB'), ('fast', 'JJ')]

Reading the Trellis

The diagram below shows the same computation as a Viterbi trellis — every candidate tag at every position is a node, every legal transition is an edge carrying its probability, and the algorithm's job is to find the highest-scoring path from START to the end. The winning path is drawn solid and in green; the path a human grammarian would have preferred, RB, is drawn dashed to show it was considered and rejected, not ignored.

Beyond HMMs: What Production Taggers Use Today

The bigram HMM you just derived is the cleanest model to learn the mathematics from, but it has two structural weaknesses worth naming. First, its Markov assumption only looks one tag back — it cannot use information like "this word ends in -tion, so it's almost certainly a noun" unless that information is folded awkwardly into the tag itself. Second, it can only use the previous tag, never the word two positions ahead, even though "fast" being followed by a full stop versus by "bowler" is informative. Two later families of models fix this. Maximum Entropy Markov Models (MEMMs) and, more robustly, Conditional Random Fields (CRFs) replace the rigid emission/transition split with a single scoring function that can use arbitrary overlapping features of a word — its prefix, suffix, capitalization, neighboring words, neighboring tags — all at once, discriminatively. More recently, neural sequence taggers (bidirectional LSTMs, and now transformer-based models such as BERT fine-tuned for tagging) push per-token accuracy on well-resourced languages like English into the high 90s percent range, largely by learning rich word representations instead of relying on raw word-identity counts the way our toy HMM did. The underlying task, though — assign the correct grammatical category using context, not the word in isolation — is unchanged from the HMM formulation; only the machinery for using context has become more powerful.

POS Tagging for Hindi and Other Indian Languages

Everything above was illustrated in English, but the task is at least as important, and considerably harder, for Indian languages. Hindi has comparatively free word order — "राहुल तेज़ गेंद फेंकता है" and several reorderings of the same words remain grammatical — so an HMM's simple "previous tag predicts next tag" assumption captures much less regularity than it does in English's stricter subject-verb-object order. Hindi and most Indian languages also use postpositions (markers like को, से, में that follow the noun) rather than English-style prepositions that precede it, and rich case-marking and verb agreement that English does not have at all — a single Hindi verb form can encode tense, gender, and honorific level simultaneously. Because of this, Indian-language NLP groups — notably at IIT Bombay and IIIT Hyderabad — have built dedicated tagged corpora and adapted tagsets (including a Bureau of Indian Standards-backed common tagset effort so that Hindi, Marathi, Bengali, and other Indian languages can be tagged with a shared, linguistically appropriate label set rather than one borrowed wholesale from English grammar). If you build a chatbot or voice assistant for an Indian-language user base, this is precisely the layer that requires local training data — an English-trained tagger will consistently misjudge Hindi word order patterns it has never seen.

Where This Shows Up in Your Exams and Beyond

The Bayes'-theorem manipulation in the "Hidden Markov Model, Derived" section — flipping P(T|W) into P(W|T)·P(T) and dropping the constant denominator — is the identical algebraic move tested in JEE Main and BITSAT conditional-probability questions; only the objects being conditioned on have changed from coins and cards to words and tags. If you are doing the CBSE Artificial Intelligence practical curriculum, POS tagging is usually the first place you use Python's NLTK library (nltk.pos_tag()) as more than a black box — understanding the HMM math above is what lets you explain, in a viva or project report, why the tagger occasionally gets an ambiguous word wrong, rather than treating a wrong tag as an unexplained bug. And at Olympiad or KVPY level, the Viterbi algorithm itself is a genuinely good example of dynamic programming with an optimal-substructure argument: the fact that you only need to keep the single best path into each node, discarding all worse paths, is exactly the kind of "why does greedy-looking pruning still give the global optimum" argument that appears in algorithmic-thinking olympiad problems, just dressed in NLP clothing instead of a graph-shortest-path story.

Check Your Understanding

• 1. Tag every word in "Bowlers bowl the fast ball" using the Penn Treebank tags NNS, VBP, DT, JJ, NN. (Hint: "bowlers" is a plural noun and "bowl" is the verb agreeing with it. Unlike "Rahul bowls fast" from this chapter's worked example — where a fluent speaker reads "fast" as an adverb describing how someone bowls — here "fast" sits directly before the noun "ball" and describes the ball itself, so its correct role is unambiguously adjectival.)
• 2. Using the transition counts given, show that P(JJ | DT) = 0 in this toy corpus (in every training sentence, DT is followed only by NN). Now consider tagging "The fast bowler." Explain why a plain HMM assigns this entire sentence a probability of exactly zero the moment it needs a DT-to-JJ transition, no matter how good every other transition and emission in the sentence is, and name one standard technique real taggers use to stop a single unseen transition from zeroing out a whole sequence.
• 3. Construct one English sentence where the word "close" is a verb, and one where it is an adjective. Identify which neighboring word in each sentence is the strongest signal that lets a reader disambiguate it.
• 4. Add a new training sentence, "Kohli/NN bowls/VB fast/RB", to the six-sentence corpus (this is a genuinely new sentence — "Kohli" and this exact wording appear nowhere in the original six). Recompute P(RB|VB), P(JJ|VB), P(fast|RB), and P(fast|JJ) for the enlarged seven-sentence corpus, then redo the Position 3 calculation for the test sentence "Rahul bowls fast." Show that this single extra RB-tagged occurrence of "fast" brings Score(RB) and Score(JJ) to an exact tie, and state in one sentence what this tipping point demonstrates about how little extra data it can take to overturn a skewed statistic.
• 5. Explain in your own words why P(T|W) cannot be estimated directly from a corpus of reasonable size, forcing the use of Bayes' theorem to rewrite it as P(W|T)·P(T).

Summary

Part-of-speech tagging assigns each word in a sentence a grammatical category, and the entire difficulty of the task comes from the fact that most frequent words are legitimately ambiguous across categories — the correct tag depends on context, never on the word alone, a point the "I saw her duck" and "read/read" examples were chosen to make unmissable. Tagsets range from the compact, cross-lingual Universal POS set to the finer, English-specific 36-tag Penn Treebank set. The Hidden Markov Model formalizes tagging as choosing the tag sequence T that maximizes P(w_i|t_i)·P(t_i|t_i-1) multiplied across the sentence — a direct application of Bayes' theorem plus two simplifying independence assumptions — and the Viterbi algorithm finds that maximizing sequence efficiently using dynamic programming. The worked toy example showed both how to compute this by hand and a genuinely important limitation: an HMM tagger's answer is only as good as the statistics of its training corpus, which is why real systems train on far larger, more carefully balanced tagged corpora, and why modern taggers (CRFs, neural sequence models) add richer context and features on top of the same underlying idea. For Indian languages, the same task is complicated further by free word order and postpositional, richly inflected morphology, which is why dedicated Indian-language tagged corpora and tagsets exist rather than simply reusing English-built tools.