A Hindi Message That Turns Into Garbage
Imagine a shopkeeper in Lucknow sends a WhatsApp Business message to a customer: "नमस्ते, आपका ऑर्डर तैयार है" (Hello, your order is ready). The customer opens it on an old billing software that was never updated, and instead of Hindi words, the screen shows something like "नमसà¥à¤¤à¥‡, आपका". Nothing about the message was corrupted in transit — no bit was flipped, no file was damaged. Yet the words have turned into meaningless symbols. This strange effect even has a name among programmers: mojibake, from Japanese for "character transformation."
By the end of this chapter you will be able to explain, precisely and mathematically, why this happens — not as a vague "computer glitch" but as a predictable, traceable consequence of how text is turned into numbers, and what happens when two programs disagree about which numbering system is being used. That disagreement is the entire story of ASCII and Unicode.
Computers Only Ever Store Numbers
A computer's memory is built from billions of tiny switches that can only be in one of two states — on or off, 1 or 0. There is no switch setting for "the letter A" or "the Devanagari letter अ." So before any piece of text can be stored, moved across the internet, or saved to a file, every single character must first be assigned a number. Only numbers travel through wires and sit in memory chips.
Think of a school's attendance register that, instead of writing full names every day, uses roll numbers — roll number 14 always refers to the same student, and the class teacher keeps a master list mapping roll numbers to names. A computer's character encoding works exactly like that master list: it is a fixed rulebook that assigns one unique number to every character you might want to type. When you press the "A" key, the keyboard driver looks up "A" in the rulebook, finds its assigned number, and that number — not a picture of the letter — is what actually gets stored in memory. The letter shape you see on your screen is drawn afterward, on demand, by a font, which uses that stored number as a lookup key to decide which shape to paint. This is worth stating as a rule you should never forget: the number is what the computer stores; the glyph you see is only ever a rendering of that number, produced fresh every time.
ASCII: The First Widely Agreed Rulebook
The first major standard rulebook of this kind was ASCII — the American Standard Code for Information Interchange. It was developed by an American standards committee and first published in 1963, with the version still referenced today finalized by 1968. ASCII assigns every character a whole number from 0 to 127. Since 127 is the largest number you can represent using 7 binary digits (2⁷ = 128 possible values, numbered 0 through 127), ASCII is often called a "7-bit" code. In practice, computers store each ASCII character in a full 8-bit byte anyway, with the extra leading bit simply set to 0.
Here is a slice of the ASCII table you should know cold, because the rest of this chapter builds directly on it:
Character Decimal 8-bit binary
space 32 00100000
0 48 00110000
9 57 00111001
A 65 01000001
Z 90 01011010
a 97 01100001
z 122 01111010
Let's put this to work on a real worked example: encoding the two-letter word "AI." Looking up the table, A = 65 and I = 73 (I is two letters after G=71... more directly, since A=65 and the alphabet is sequential, I is the 9th letter, so I = 65 + 8 = 73). Converting each to 8-bit binary:
A = 65 = 01000001
I = 73 = 01001001
So the two characters "AI" are stored in memory, in order, as the two bytes 01000001 01001001. Nothing else is stored — no quotation marks, no indication that these bytes represent "text" rather than "a small number." The computer trusts whatever program reads those bytes later to know they should be interpreted as ASCII characters. In Python, you can see this mapping directly using two built-in functions, ord() (character to number) and chr() (number to character):
>>> ord('A')
65
>>> ord('I')
73
>>> chr(65)
'A'
>>> bin(ord('A'))
'0b1000001'
A Deliberate Pattern in the Table — and a Misconception to Correct
Look again at the numbers: A=65, a=97. The gap is exactly 32. This is true for every matching letter pair — B=66/b=98, Z=90/z=122 — because the designers of ASCII placed uppercase and lowercase letters exactly 32 apart on purpose, so that flipping a single bit (bit value 32, i.e. the 6th binary digit) toggles a letter between uppercase and lowercase. This is precisely how case-conversion functions work internally:
>>> chr(ord('m') - 32)
'M'
Here is a misconception worth correcting explicitly, because many students carry it by accident: "A" and "a" are the same character to a computer, just displayed differently. This is false. They are two completely different, unrelated numbers (65 and 97) that happen to sit near each other in the table by deliberate design. A computer comparing "Apple" and "apple" character-by-character will find they differ starting at the very first letter, because 65 ≠ 97 — unless a programmer explicitly converts both to the same case first.
This same fact produces a second, more surprising trap. Because digits (48–57) come before uppercase letters (65–90), which come before lowercase letters (97–122), sorting text by raw character code does not match the alphabetical order you expect from a dictionary. Since Z = 90 and a = 97, and 90 is less than 97, the word "Zebra" is sorted before "apple" in a plain code-based sort — even though a human alphabetizing by eye would put "apple" first. This is exactly why spreadsheet and database software offer a separate "case-insensitive" sort option — the raw numeric order and the human alphabetical order are genuinely different things.
Where 128 Slots Run Out
ASCII's 128 numbers are just enough for the 26 English letters (upper and lower case), the 10 digits, common punctuation, and a handful of control codes (like 10 for a new line). There is exactly zero room left over for Devanagari, Tamil, Telugu, Bengali, Gurmukhi, Kannada, Malayalam, Gujarati, Odia, Chinese, Arabic, or even accented European letters. ASCII was built for English text and nothing else.
The first fix engineers tried was to use the unused 8th bit of a byte to reach numbers 128–255, doubling the table to 256 slots — commonly called "extended ASCII." The problem: every hardware maker, software company, and country invented its own, mutually incompatible mapping for that upper half. A byte with value 200 might mean one accented letter under one vendor's table and a completely different Indian-script character under another's. There was no single universal agreement — only a patchwork of regional "code pages."
India's own real historical solution was ISCII — the Indian Script Code for Information Interchange, standardized by the Bureau of Indian Standards as IS 13194 in 1991. ISCII used the extended 8-bit range and a set of special escape codes to switch between different Indian scripts within the same document, and it was used in early Indian government and DTP (desktop publishing) systems. But ISCII shared the same fundamental weakness as every other extended-ASCII scheme: it was not compatible with the code pages used outside India, only one script could be reliably active at a time within a document, and a file created on an ISCII-aware system still turned to garbage when opened on a system expecting a different 8-bit table. This incompatibility across regional standards — Indian systems using one table, European systems using another, Japanese systems using a third — is the direct root cause of the mojibake problem from the opening of this chapter.
Unicode: One Table for Every Script on Earth
The real fix was not to invent yet another regional 256-slot table, but to throw away the 256 limit entirely and build a single global rulebook big enough for every writing system that has ever existed. This effort, begun by engineers at Xerox and Apple in the late 1980s, was formalized when the Unicode Consortium was founded in 1991, releasing Unicode 1.0 the same year.
Unicode assigns every character a number called a code point, written in a distinctive format like U+0041 (the "U+" simply marks it as a Unicode code point, followed by the number in hexadecimal). Instead of stopping at 127 or 255, Unicode reserves code points from U+0000 all the way to U+10FFFF — more than 1.1 million possible slots, of which roughly 150,000 have been assigned so far. Every major Indian script has its own dedicated, contiguous block: Devanagari occupies U+0900 to U+097F, with Tamil, Telugu, Bengali, Gurmukhi, Gujarati, Kannada, Malayalam, and Odia each holding their own separate ranges nearby. There is finally room for all of them to coexist in the same document, at the same time, without any script-switching tricks.
Here is the design decision that makes Unicode backward-compatible with everything that came before: Unicode's first 128 code points, U+0000 through U+007F, are numerically identical to the ASCII table. The letter A is code point U+0041, which is 65 in decimal — the exact same number ASCII already used. This was deliberate: every plain-English ASCII document ever created automatically counts as valid Unicode text too, with no conversion required.
Code Points Are Not Bytes — Meet the Encoding
A Unicode code point is only an abstract number, similar to an entry in a phone book — a name mapped to a number, nothing about how that number gets sent over a wire. To actually store or transmit a code point, it must be converted into a specific sequence of bytes, and the rule for doing that conversion is called an encoding. Unicode defines several valid encodings — UTF-8, UTF-16, and UTF-32 — and this is exactly the distinction behind our second misconception to correct: Unicode and UTF-8 are not the same thing. Unicode is the standard that assigns the numbers; UTF-8 is one particular scheme for packing those numbers into bytes. The same Devanagari letter has one fixed Unicode code point, but its byte representation differs completely depending on which encoding is chosen.
UTF-8, designed by Ken Thompson and Rob Pike in 1992, is by far the dominant encoding used on the modern web and on virtually every Indian government, banking, and UPI-linked website today, because of one elegant property: it is a variable-length encoding that uses only as many bytes as a character actually needs.
Code point range Bytes used Byte pattern
U+0000 – U+007F 1 0xxxxxxx
U+0080 – U+07FF 2 110xxxxx 10xxxxxx
U+0800 – U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000 – U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Notice the first row: code points U+0000 to U+007F — exactly the ASCII range — are stored in a single byte whose pattern is 0xxxxxxx, which is precisely how ASCII already stored them. This is why old ASCII text files need zero modification to also be correct UTF-8 files. Every row after that uses a "continuation byte" pattern starting with 10, and the leading byte's number of 1-bits before the first 0 tells a reader exactly how many bytes the whole character occupies. This makes UTF-8 self-synchronizing: software that lands in the middle of a byte stream can always tell, just by looking at one byte, whether it is sitting at the start of a character or in the middle of one — a genuinely clever piece of engineering.
Worked Example: Encoding अ Byte by Byte
Let's fully encode the Devanagari letter अ (the independent vowel "a," the very first letter of the Devanagari script) into UTF-8, one step at a time.
Step 1 — find the code point. अ is Unicode code point U+0905, which is 2309 in decimal (9×256 + 0×16 + 5 = 2309).
Step 2 — choose the byte pattern. 2309 falls inside the range U+0800–U+FFFF, so UTF-8 requires 3 bytes, using the template 1110xxxx 10xxxxxx 10xxxxxx, which has room for 4 + 6 + 6 = 16 data bits.
Step 3 — write 2309 as 16-bit binary.
2309 = 0000 1001 0000 0101
Step 4 — slice those 16 bits into groups of 4, 6, and 6, left to right.
0000 | 100100 | 000101
Step 5 — drop each slice into its matching byte template.
Byte 1: 1110 + 0000 = 11100000 = 0xE0
Byte 2: 10 + 100100 = 10100100 = 0xA4
Byte 3: 10 + 000101 = 10000101 = 0x85
So the single character अ is physically stored as three bytes: E0 A4 85. You can check this instantly in Python, and it also reveals the crucial distinction between counting characters and counting bytes:
>>> "अ".encode('utf-8')
b'\xe0\xa4\x85'
>>> len("अ")
1
>>> len("अ".encode('utf-8'))
3
Python's len() on a string counts characters (code points), giving 1. But the actual storage on disk or over a network uses 3 bytes. One visible character is not one byte — this gap is the single most important practical fact in this whole chapter, and it is illustrated below.
Why This Gap Matters in Real Life
This one-character-can-be-many-bytes fact is not just theoretical. It shows up in three places any Indian student will recognize:
SMS length limits. A standard SMS sent using the classic GSM 7-bit alphabet (a compact encoding closely related in spirit to ASCII, covering basic Latin text) allows up to 160 characters in a single message segment. The instant a message contains even one character outside that small alphabet — a single emoji, a rupee sign typed in certain fonts, or any Hindi, Tamil, or other Indian-script word — telecom networks switch the entire message to UCS-2, a fixed-width 16-bit Unicode encoding, and the limit per segment drops to 70 characters. This is exactly why an English-only OTP message fits comfortably in one segment while a Hindi bank alert of similar visible length often splits into two message segments — a direct, practical consequence of encoding choice, not message content length in any everyday sense.
File size and mobile data. Because most Indian-script characters need 3 bytes each in UTF-8 while English letters need only 1, a Hindi or Tamil news article of similar length to an English one takes roughly three times the storage and data bandwidth in UTF-8. This has genuinely shaped decisions by Indian apps and government portals serving users on limited mobile data plans, favouring efficient text delivery.
Mojibake, fully explained. Return now to the opening scenario. A UTF-8 file storing "नमस्ते" holds a specific sequence of multi-byte characters — each byte like E0, A4, or 85 individually means nothing on its own; they only make sense grouped in threes as UTF-8. If a program wrongly assumes those same bytes are old single-byte extended-ASCII text (where every byte is read as one independent character), it will read E0 as one character, A4 as an unrelated second character, and 85 as a third — producing exactly the "नमसà¥à¤¤à¥‡"-style garbage from the start of this chapter. Nothing was corrupted; the sender wrote bytes meaning one thing, and the reader interpreted the identical bytes meaning something else entirely. Mojibake is a completely deterministic, traceable mismatch between two encoding assumptions — never random noise.
A Third Misconception, Corrected
One more idea worth explicitly ruling out: "Unicode is a font." It is not. Unicode only fixes the number-to-meaning mapping — U+0905 will always mean "the Devanagari letter अ" on every properly built system on earth. But the exact pixel shape used to draw that letter is chosen independently by whichever font is installed, which is why the same emoji or the same Devanagari letter can look slightly different across two phones or two apps while still being, underneath, the identical Unicode code point.
Practice: Active Recall
- Using the table earlier in this chapter, what is the ASCII decimal value of the character '5'? Write it as 8-bit binary.
- Work out
chr(ord('m') - 32)by hand, without running code. What letter results, and why does subtracting 32 produce it? - The Devanagari letter न (na) is Unicode code point U+0928, decimal 2344. Using the same 3-byte UTF-8 method worked through for अ, encode न into its three UTF-8 bytes. Show your 16-bit binary before converting to hex.
- A text file containing only the English word "Delhi" is saved once using ASCII and once using UTF-8. Using what you learned about Unicode's first 128 code points, explain why the two saved files are byte-for-byte identical.
- In your own words, explain why an all-English SMS can hold up to 160 characters while a Hindi SMS of similar visible length is often sent as two separate message segments.
Answer key: (1) '5' = 53 = 00110101. (2) 'm' = 109; 109 − 32 = 77 = 'M', because the ASCII table places every lowercase letter exactly 32 above its matching uppercase letter. (3) 2344 in 16-bit binary is 0000 1001 0010 1000; sliced into 4+6+6 as 0000 | 100100 | 101000, giving bytes 11100000 (0xE0), 10100100 (0xA4), 10101000 (0xA8) — so न is stored as E0 A4 A8. (4) Because Unicode's first 128 code points (U+0000–U+007F) are defined to be numerically identical to ASCII, and every letter in "Delhi" falls in that shared range, both encodings produce the exact same single-byte-per-letter sequence. (5) Adding any character outside the basic GSM 7-bit alphabet — which includes essentially all Hindi text — forces the entire message into 16-bit UCS-2 encoding, cutting the per-segment character limit from 160 down to 70, so the same visible length of Hindi text needs more segments than the equivalent English text.
Summary
Computers store only numbers, never pictures of letters, so every character system begins with a fixed table mapping characters to numbers — a character encoding. ASCII, standardized by 1968, mapped 128 characters (enough only for English) to numbers 0–127, with a deliberate 32-value gap between matching upper and lower case letters. ASCII's 128 slots could not hold Indian scripts, so India built its own extended 8-bit standard, ISCII, in 1991 — but incompatible regional extensions like this, each disagreeing on what the upper 128 byte values meant, are precisely what caused (and still cause) mojibake. Unicode, formalized the same year by the Unicode Consortium, solved this by defining a single global table of over a million possible code points, deliberately keeping its first 128 values identical to ASCII for backward compatibility, and giving every world script — including all major Indian scripts — its own dedicated block. A code point is only an abstract number; UTF-8, the dominant encoding today, converts each code point into 1 to 4 bytes using a self-synchronizing pattern, storing ASCII characters unchanged in a single byte while most Indian-script characters need three. That gap between character count and byte count directly explains real phenomena Indian students encounter daily: shorter Hindi SMS limits, larger Hindi file sizes, and the exact mechanism behind mojibake — a fully deterministic mismatch between the encoding used to write bytes and the encoding assumed while reading them.