Tokenization: BPE, WordPiece, and SentencePiece

Type "The IRCTC website is unreachable" into ChatGPT and something invisible happens before the model thinks even a single thought: the sentence is chopped into pieces. Not words, not letters — pieces. The model never actually sees the letter "I" or the word "unreachable". It sees a list of integers, something like [464, 41575, 4177, 3052, 318, 555, 16250, 540]. Every idea a language model has ever expressed began as a lookup in a table of these pieces, called tokens. This chapter is about the surprisingly deep question of how you cut text into tokens — and why the three dominant answers (BPE, WordPiece, SentencePiece) are cleverer than they first appear.

Why not just use words? The vocabulary trap

The obvious idea is: one token per word. Split on spaces, build a dictionary, done. This fails badly, and understanding why motivates everything that follows.

First, the vocabulary explodes. English alone has hundreds of thousands of words, and a multilingual model trained on Hindi, Tamil, and English together would need millions. Every token needs its own row in a giant embedding matrix, so a huge vocabulary means a huge, slow, memory-hungry model.

Second — and this is the killer — you can never cover every word. Language is open. New words appear constantly: "cryptowinter", "Chandrayaan", a friend's username, a typo like "unreacheble". Any word not in your dictionary becomes the dreaded <UNK> (unknown) token, and the model goes blind. Faced with "Chandrayaan-3 launched", a word-level model might see "<UNK>-3 launched" and lose the most important word in the sentence.

The opposite extreme — one token per character — never has an unknown token, since every string is made of characters. But now the model must reassemble meaning from tiny fragments. The word "understanding" becomes 13 separate steps, and sequences get so long that training and inference crawl. Attention cost grows with the square of sequence length, so quadrupling your sequence length roughly multiplies compute by sixteen.

So we are trapped between two bad options: words (small sequences, giant vocabulary, breaks on new words) and characters (tiny vocabulary, never breaks, but painfully long sequences). Subword tokenization is the escape. The idea: keep common words whole ("the", "launch"), but split rare words into reusable fragments ("Chandra" + "yaan", "un" + "reach" + "able"). Common things stay short; rare things stay expressible. Nothing is ever truly unknown, because in the worst case you fall back to individual characters.

Byte-Pair Encoding: compression turned into learning

Byte-Pair Encoding (BPE) is the most widely used subword algorithm — it powers GPT-2, GPT-3, GPT-4, and LLaMA's tokenizer. It was originally a 1994 data-compression trick by Philip Gage, repurposed for NLP by Sennrich, Haddow and Birch in 2016. The core insight is beautifully simple: start from characters, then repeatedly glue together the pair that occurs most often.

Let me walk through the actual algorithm on a tiny training corpus. Suppose after counting our text we have these four "words" with their frequencies (the </w> marker means "end of word", so the tokenizer knows where words stop):

low     </w>              (appears 5 times)
low e r </w>            (appears 2 times)
new e s t </w>          (appears 6 times)
wide s t </w>           (appears 3 times)

We begin with every word split into individual characters:

(l o w </w>)        : 5
(l o w e r </w>)    : 2
(n e w e s t </w>)  : 6
(w i d e s t </w>)  : 3

Step 1 — count every adjacent pair, weighted by word frequency. The pair (e, s) appears in "newest" (6 times) and "widest" (3 times), giving a count of 9. Let us check a competitor: (l, o) appears in "low" (5) and "lower" (2), count 7. Since 9 > 7, the winner is (e, s). We merge it everywhere into a single new token es:

(l o w </w>)         : 5
(l o w e r </w>)     : 2
(n e w es t </w>)    : 6
(w i d es t </w>)    : 3

Step 2 — recount. Now (es, t) appears 6 + 3 = 9 times. Merge into est. Step 3 — (est, </w>) appears 9 times; merge into est</w>. Step 4 — now (l, o) wins with count 7; merge into lo. Step 5 — (lo, w) has count 7; merge into low. Step 6 — (n, e), count 6, merge into ne. After six merges our corpus looks like this:

(low </w>)           : 5
(low e r </w>)       : 2
(ne w est</w>)       : 6
(w i d est</w>)      : 3

I verified this exact trace by running the algorithm; the six merges learned, in order, are: e+s, es+t, est+</w>, l+o, lo+w, n+e. That ordered list of merge rules IS the trained tokenizer. That is the whole model. Real BPE simply repeats this until the vocabulary reaches a target size — GPT-2 stops at about 50,257 tokens.

The diagram below shows these merges as a tree growing upward from characters to subwords — this is the mental model to keep.

Applying a trained BPE tokenizer to new text

Training gave us an ordered rule list. To tokenize a new word, split it into characters and apply the merge rules in the order they were learned, greedily. Take the word "lowest", which never appeared in training. Start as l o w e s t </w> and replay our six rules:

1. Rule e+s: → l o w es t </w>
2. Rule es+t: → l o w est </w>
3. Rule est+</w>: → l o w est</w>
4. Rule l+o: → lo w est</w>
5. Rule lo+w: → low est</w>
6. Rule n+e: no match.

Final tokens: ["low", "est</w>"] — two tokens for a word the tokenizer had literally never seen, and both pieces are meaningful reusable units. That is the magic of subwords: generalization to unseen words with no <UNK> ever needed.

WordPiece: merge by usefulness, not just frequency

WordPiece, developed at Google and used by BERT, is BPE's close cousin with one crucial change. BPE merges the most frequent pair. WordPiece asks a smarter question: which merge best explains the data? Instead of raw count, it picks the pair that maximizes a likelihood score. Concretely, for a candidate pair of tokens \(a\) and \(b\), WordPiece scores:

score(a, b) = count(a b) / ( count(a) × count(b) )

Read this carefully. The numerator rewards a pair that appears often. The denominator penalizes pieces that are already very common on their own. Consider the letters "e" and "s" in English: both are individually super-common, so count(e) and count(s) are huge, which shrinks the score even if "es" is frequent. But a pair like "ch" + "ai" (as in "Chai" or "Chennai") might appear together far more often than their individual popularity would predict — a high ratio — so WordPiece merges it eagerly. The score, in effect, measures how much a pair "belongs together" beyond chance. This is exactly the idea of pointwise mutual information: merge the pair whose togetherness is most surprising.

There is also a visible surface difference. WordPiece marks continuation pieces with ## instead of marking word ends. BERT tokenizes "playing" as ["play", "##ing"] and "Chandrayaan" might become ["Chan", "##dra", "##ya", "##an"]. The ## tells the model "this piece attaches to the previous one, not a new word." At inference time, WordPiece uses greedy longest-match-first: it grabs the longest prefix in the vocabulary, emits it, then repeats on the remainder.

SentencePiece: throw away the assumption of spaces

Both algorithms above quietly assume you can pre-split text on spaces to get "words". For English that is fine. But this assumption breaks for much of the world. Written Japanese and Chinese use no spaces at all. And even in Indian languages, space handling is fragile once you mix scripts. Worse, if the tokenizer strips spaces before processing, it cannot perfectly reconstruct the original text — is it "New Delhi" or "NewDelhi"? — which matters when a model must generate output character-for-character.

SentencePiece, from Google, solves this by treating the input as a raw stream of Unicode characters, spaces included. It makes one elegant move: replace every space with a visible marker ▁ (U+2581, "lower one eighth block") before tokenizing. Now the space is just another character the algorithm can merge or split. "New Delhi" becomes the character stream ▁New▁Delhi, and the tokenizer might produce ["▁New", "▁Del", "hi"]. Because the space marker is preserved inside the tokens, detokenizing is trivial and lossless: concatenate the tokens and turn every ▁ back into a space. This property is called being reversible, and it is why SentencePiece is used by T5, ALBERT, XLNet, and the LLaMA family.

A key point students confuse: SentencePiece is not a fourth competing algorithm at the same level as BPE and WordPiece. It is a library and framework that can run either the BPE merge rule or a different "unigram" model underneath. Its real contribution is the language-agnostic, space-as-a-character, fully reversible design — not a new merge criterion.

Common misconception: "tokens are words" (and why the API bill proves otherwise)

The single most common mistake at this level is believing one token equals one word. It does not. For typical English, one word averages roughly 1.3 tokens, because common words are one token but longer or rarer words split. Numbers, code, emoji, and non-Latin scripts fare worse. A Hindi sentence in Devanagari often uses several tokens per word, because these tokenizers were trained on English-heavy corpora and never learned long Hindi subwords — so Hindi text gets shredded closer to the character level.

This has a real cost. LLM APIs charge per token. If a Hindi paragraph uses three times as many tokens as the same meaning in English, an Indian startup building a Hindi chatbot pays roughly three times more for the same conversation and hits the model's context-length limit three times faster. This is not a rounding detail — it is an active fairness and cost issue in multilingual AI, and it flows directly from how the tokenizer was trained. Understanding tokenization is understanding why your API bill and your context window behave the way they do.

Comparing the three at a glance

• BPE — merges the most frequent adjacent pair. Simple, fast, greedy. Used by GPT-2/3/4, RoBERTa, LLaMA. Marks word boundaries (e.g. </w> or a leading space byte).
• WordPiece — merges the pair with the highest likelihood ratio count(ab)/(count(a)count(b)), favouring pieces that belong together beyond chance. Used by BERT, DistilBERT. Marks continuations with ##.
• SentencePiece — a framework, not a merge rule. Treats raw text (spaces included, as ▁) as the input, so it is language-agnostic and losslessly reversible. Runs BPE or unigram underneath. Used by T5, ALBERT, LLaMA.

Worked exam-style problem

Problem. A BPE tokenizer is trained on a corpus where the only relevant words are bat (frequency 4), batting (frequency 3), and batsman (frequency 2). Starting from characters (ignore the end-of-word marker for brevity), which pair is merged first, and why?

Solution. Count each adjacent pair weighted by word frequency. The pair (b, a) appears in all three words: 4 + 3 + 2 = 9. The pair (a, t) also appears in all three: 4 + 3 + 2 = 9. Every other pair (like (t, t) in "batting" = 3, or (t, s) in "batsman" = 2) is smaller. So we have a tie between (b,a) and (a,t) at count 9. BPE breaks ties by a fixed deterministic rule (typically the first encountered / lexicographic order), so the implementation would merge one of them — say (b,a) → ba — and on the next round (ba, t) would win with count 9, giving the highly reusable token bat. The lesson: BPE naturally discovers the shared stem "bat" because it is the most frequent overlapping fragment, exactly the behaviour we want.

Active recall — do these yourself

1. Using the six merge rules learned in the main example (e+s, es+t, est+</w>, l+o, lo+w, n+e), tokenize the new word "newer" step by step. (Hint: start n e w e r </w> and apply rules in order — which rules fire?)
2. Explain in one sentence why a pure character-level tokenizer never produces <UNK>, yet is rarely used for large models anyway.
3. WordPiece would be reluctant to merge two individually very common letters even if they often appear side by side. Point to the exact term in the score formula count(ab)/(count(a)count(b)) that causes this reluctance.
4. Your friend claims "SentencePiece is just a faster version of BPE." Correct them in two sentences.
5. A Hindi chatbot costs 3× more per message than an English one on the same model. Give the tokenization reason, and one thing a team could do about it (hint: think about who trained the tokenizer and on what).

Answer to Q1: n e w e r </w> → rule n+e fires → ne w e r </w>; no other rule matches (there is no e s, no l o, etc.), so the final tokenization is ["ne", "w", "e", "r", "</w>"] — a good reminder that a word can still fragment heavily if its pieces were not common in training.

Summary — the key ideas

Tokenization is the bridge between human text and the integer world a neural network lives in, and the whole field is a negotiation between two failure modes: word-level tokenizers have giant vocabularies and break on unseen words, while character-level tokenizers never break but produce crushingly long sequences. Subword tokenization threads the needle by keeping common words whole and splitting rare words into reusable fragments, so nothing is ever truly unknown. BPE learns this by greedily merging the most frequent adjacent pair, producing an ordered list of merge rules that is the trained tokenizer. WordPiece swaps raw frequency for a likelihood ratio, merging pieces that belong together more than chance would predict, and marks continuations with ##. SentencePiece is not a rival merge rule but a language-agnostic, space-preserving, losslessly reversible framework that runs BPE or unigram underneath — the design that makes tokenization work for scripts without spaces. And the whole subject has a sharp practical edge: because these tokenizers were trained on English-heavy text, Indian-language input costs more tokens, more money, and more context window — which is exactly why understanding tokenization matters to anyone building AI for India.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where tokenization: bpe, wordpiece, and sentencepiece is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting tokenization: bpe, wordpiece, and sentencepiece to at least 3 other topics you have studied.