AI Capstone Project: Indian Language Detector

Open WhatsApp on almost any Indian phone and you will find sentences like this one: "kal mera exam hai, thoड़ा nervous hoon." A message like "yeh bahut accha hai" sits two lines below "The weather is very nice today." Both lines use the same twenty-six letters of the Latin alphabet. Yet one is Hindi written in Roman script — commonly called Hinglish — and the other is English. Every keyboard app on your phone, every customer-support chatbot for an Indian bank, and every browser's "Translate this page?" popup has to solve exactly this problem thousands of times a second: given a short piece of text, which language is it in?

This capstone builds a real, working Indian Language Detector in Python. It is small enough to trace by hand, line by line, yet it uses the same core idea — learning statistical patterns from labelled examples — that powers production language-identification systems used by browsers and translation apps. Along the way, you will bring together everything a machine learning project needs: a problem definition, training data, a feature representation, a model, and an evaluation step. This mirrors the AI Project Cycle taught in the CBSE AI curriculum — Problem Scoping, Data Acquisition, Data Exploration, Modelling, and Evaluation — and it is worth noticing, as we go, which part of the project each section belongs to.

Attempt 1: Just Look at the Alphabet

The first idea anyone has is the simplest one: different Indian languages are usually written in different scripts. Hindi uses Devanagari (यह अच्छा है). Tamil uses the Tamil script (இது நல்லது). English uses the Latin alphabet (this is good). Each script occupies its own reserved range of numeric codes inside Unicode, the standard that assigns every character in every writing system a unique number. Devanagari characters live between code point 0x0900 and 0x097F; Tamil characters live between 0x0B80 and 0x0BFF. Python's built-in ord() function gives you the numeric code point of any character, so checking the script a piece of text is written in is a one-line comparison per character.

def script_of(char):
    code = ord(char)
    if 0x0900 <= code <= 0x097F:
        return "devanagari"
    if 0x0B80 <= code <= 0x0BFF:
        return "tamil"
    if char.isalpha():
        return "latin"
    return "other"

def detect_script(text):
    counts = {"devanagari": 0, "tamil": 0, "latin": 0}
    for ch in text:
        s = script_of(ch)
        if s in counts:
            counts[s] += 1
    best_script = max(counts, key=counts.get)
    if counts[best_script] == 0:
        return "unknown"
    return best_script

Trace this by hand on the Hindi sentence "यह अच्छा है". Every character in it is a Devanagari letter or a Devanagari vowel sign, so the loop increments counts["devanagari"] on every pass and leaves "tamil" and "latin" at zero. max(counts, key=counts.get) asks Python for the dictionary key whose value is largest, which is "devanagari". The function correctly returns "devanagari". Run it on "இது நல்லது" and by the identical reasoning it returns "tamil". For text written in a native Indian script, checking the alphabet works perfectly, and it costs almost nothing to compute — no training data required at all.

Where Script-Checking Breaks: Hinglish

Now run detect_script on "kal mera exam hai". Every character is a plain Latin letter, so counts["latin"] climbs and the function correctly reports "latin". But "latin script" is not an answer to the question we actually care about — it does not tell us whether the sentence is Hindi or English. "The weather is very nice" is also entirely Latin script, and produces the exact same "latin" verdict. Script-checking cannot tell these two sentences apart, because it never looks at anything except which alphabet is being used, and both sentences use the same one.

This is worth stating as a general lesson, because it is a mistake beginners make constantly when building any classifier: a feature that is very useful for separating some classes can be completely useless for separating others. Script tells Hindi-in-Devanagari apart from English immediately. It tells Hindi-in-Roman-script (Hinglish) apart from English not at all. A detector needs a second, different kind of evidence for the case where the first kind runs out — which is exactly what a statistical model built from real examples of each language provides.

Turning Words Into Evidence: A Word-Frequency Classifier

The idea behind the fix is one you already use without realising it. If you saw a message containing the word "hai" or "nahi" or "accha", you would guess Hindi immediately, even without seeing the rest of the sentence — because you have read thousands of Hindi and English sentences before, and you have learned, informally, which words tend to appear in which. A machine learning classifier formalises exactly this intuition: instead of guessing from a lifetime of reading, it counts word frequencies from a small labelled dataset called training data, and uses those counts as its evidence.

Start with five short example sentences for each of the two classes we need to separate, English and Hinglish. This is the Data Acquisition stage of the project — deliberately tiny here so every count can be checked by hand, though a production system would use thousands of sentences.

english_training = [
    "I am going to school today",
    "The weather is very nice",
    "She is my best friend",
    "We are watching a movie",
    "This is a good book",
]

hinglish_training = [
    "kal mera exam hai",
    "yeh bahut accha hai",
    "tum kaise ho aaj",
    "mujhe bhookh lagi hai",
    "woh mera dost hai",
]

The feature we will extract from each sentence is simply its set of words, lower-cased and split on spaces. The model is nothing more than a dictionary counting how many times each word appeared across a class's training sentences — this is the Data Exploration and Modelling stage.

def build_word_counts(sentences):
    counts = {}
    for sentence in sentences:
        for word in sentence.lower().split():
            counts[word] = counts.get(word, 0) + 1
    return counts

english_counts = build_word_counts(english_training)
hinglish_counts = build_word_counts(hinglish_training)

Trace build_word_counts on hinglish_training by hand, sentence by sentence. "kal mera exam hai" contributes one count each to kal, mera, exam, hai. "yeh bahut accha hai" contributes one each to yeh, bahut, accha, and a second count to hai, bringing it to 2. "tum kaise ho aaj" adds tum, kaise, ho, aaj. "mujhe bhookh lagi hai" adds mujhe, bhookh, lagi, and a third count for hai, now 3. "woh mera dost hai" adds woh, a second count for mera (now 2), dost, and a fourth count for hai, now 4. So after all five sentences, hinglish_counts["hai"] == 4 and hinglish_counts["mera"] == 2, because those two words each appeared in four and two of the five training sentences respectively. Every other Hinglish word in this tiny dataset appears exactly once. Running the same trace on english_training gives english_counts["is"] == 3 (it appears in the weather, friend, and book sentences) and english_counts["a"] == 2 (in the movie and book sentences), with every other word appearing once.

Now define a score for a new, unseen sentence: for each word in it, add up how many times that word appeared in a class's training data. In algebraic notation, for a language L and a piece of text made of words w1, w2, ..., wn:

score_L(text) = count(w1 in L) + count(w2 in L) + ... + count(wn in L)

def score(text, word_counts):
    total = 0
    for word in text.lower().split():
        total += word_counts.get(word, 0)
    return total

Worked Example: Classifying by Hand

Take the test sentence "aaj mera mood accha hai" (roughly, "today my mood is good"). Split it into five words: aaj, mera, mood, accha, hai. Compute its English score first, word by word, by looking each one up in english_counts: aaj never appeared in any English training sentence, so it contributes 0. Neither did mera (0), mood (0), accha (0), nor hai (0). The English score is 0 + 0 + 0 + 0 + 0 = 0.

Now compute the Hinglish score using hinglish_counts: aaj appeared once, contributing 1. mera appeared twice, contributing 2. mood never appeared in the Hinglish training sentences either (it is an English loanword the training data happened not to include), contributing 0. accha appeared once, contributing 1. hai appeared four times, contributing 4. The Hinglish score is 1 + 2 + 0 + 1 + 4 = 8.

English scores 0, Hinglish scores 8. Even though one word out of five (mood) matched nothing at all in either training set, the other four words carried enough evidence for the model to be confident: this sentence is Hinglish. Notice what actually did the work — not a single "magic" word, but the accumulated weight of several words, several of which are common function words (mera = "my", hai = "is/are") that appear constantly in ordinary Hindi speech and almost never in English sentences at all.

A Subtle Bug: When Both Scores Are Zero

Here is a natural way to turn the two scores into a decision:

def classify_latin_text_BUGGY(text):
    e_score = score(text, english_counts)
    h_score = score(text, hinglish_counts)
    return "english" if e_score > h_score else "hinglish"

Trace this on an English sentence built entirely from words the tiny training set never saw, such as "Cats sleep on soft blankets". None of cats, sleep, on, soft, blankets appear in english_counts or in hinglish_counts, so both e_score and h_score equal 0. The condition e_score > h_score is 0 > 0, which is False, so the else branch runs and the function confidently reports "hinglish" — for a sentence that is obviously plain English and has nothing whatsoever to do with Hindi.

This is exactly the kind of bug that is easy to miss because the code runs without crashing and even looks reasonable most of the time. The mistake here is a common misconception worth naming directly: a machine learning classifier that outputs a label is not the same as a classifier that has evidence for that label. A score of zero on both sides is not weak evidence for one side or the other — it is no evidence at all, usually meaning the input contains words the model has simply never seen (this is called being "out of vocabulary"). Treating "no evidence" the same as "weak evidence for Hinglish" is a design flaw, not a rounding error, and it comes directly from the careless choice of > as the deciding comparison. The fix is to check for the tie explicitly and say so honestly:

def classify_latin_text(text):
    e_score = score(text, english_counts)
    h_score = score(text, hinglish_counts)
    if e_score == 0 and h_score == 0:
        return "unknown"
    return "english" if e_score > h_score else "hinglish"

Re-trace "Cats sleep on soft blankets" through the corrected function: e_score = 0, h_score = 0, the new if condition is True, and the function returns "unknown" instead of guessing wrong. This is a small change, but it reflects a real engineering principle used throughout machine learning: a model should be able to say "I don't have enough information" rather than being forced to output one of a fixed list of labels every single time.

Combining Both Stages: The Full Detector

The complete Indian Language Detector chains the two techniques together. Script-checking runs first, because it is cheap and completely reliable whenever the text is written in a distinct native script. Only when the script is Latin — the one case where script alone is ambiguous between English and Hinglish — does the word-frequency classifier get invoked.

def detect_language(text):
    script = detect_script(text)
    if script == "devanagari":
        return "Hindi (Devanagari script)"
    if script == "tamil":
        return "Tamil"
    if script == "latin":
        return classify_latin_text(text)
    return "unknown"

Trace three calls. detect_language("यह अच्छा है"): detect_script returns "devanagari", so the function returns "Hindi (Devanagari script)" immediately, never touching the word-count classifier at all. detect_language("இது நல்லது"): detect_script returns "tamil", so the function returns "Tamil". detect_language("aaj mera mood accha hai"): detect_script returns "latin", so control passes to classify_latin_text, which we already traced by hand above and which returns "hinglish". Three different code paths, three correct answers, using the right tool for each situation instead of forcing one technique to do a job it is bad at.

Testing the Detector Properly

A model that has only ever been checked against its own training sentences has not really been tested — it has just had its memory examined. The Evaluation stage of the AI Project Cycle requires a separate test set: fresh sentences the model never saw while its word counts were being built, each one labelled with the correct answer in advance so we can check the model's predictions against ground truth.

Take four fresh sentences, two of each class, and run them through classify_latin_text. "The book is good" (true label: English) splits into the, book, is, good. Looking each up in english_counts: the=1, book=1, is=3, good=1, giving e_score = 6. None of those four words appear anywhere in the Hinglish training data, so h_score = 0. Since 6 > 0, the prediction is English — correct. "mera dost bahut accha hai" (true label: Hinglish) splits into mera, dost, bahut, accha, hai. In hinglish_counts: mera=2, dost=1, bahut=1, accha=1, hai=4, giving h_score = 9 against e_score = 0. Prediction: Hinglish — correct. The remaining two test sentences follow the identical procedure: "My friend is nice" (true label: English) scores e_score = 1+1+3+1 = 6 against h_score = 0, correctly predicted English; "kal exam bahut accha tha" (true label: Hinglish) scores h_score = 1+1+1+1+0 = 4 (the word tha never appeared in training, contributing 0) against e_score = 0, correctly predicted Hinglish. That gives the model 4 out of 4 on this test set: 100% accuracy.

Before celebrating, look closely at why the accuracy is so high: every single test word had actually appeared in the training data. That is not a coincidence a real deployment can rely on — it happened because the test sentences were built, deliberately, from a vocabulary of only about thirty words total. This is precisely the trap the "Cats sleep on soft blankets" example exposed earlier: a tiny, closed vocabulary makes a model look flawless on paper while leaving it unable to handle almost anything a real user actually types. A test accuracy number is only meaningful when the test data resembles the messy, unpredictable input the model will face after deployment — a rule that applies to every classifier you will ever build, not just this one.

The standard way to summarise a classifier's test results is a confusion matrix, a small table with actual labels down the rows and predicted labels across the columns.

                Predicted: English   Predicted: Hinglish
Actual: English         2                    0
Actual: Hinglish        0                    2

Both diagonal cells (correct predictions) are filled and both off-diagonal cells (mistakes) are empty, which is another way of seeing the 4/4 = 100% accuracy directly. When you later work with larger, more realistic datasets, the off-diagonal cells will not stay at zero, and reading which specific pairs of classes get confused with each other is often more informative than the single accuracy number alone.

How This Relates to Real Systems

The classifier built in this chapter is a genuine simplification of techniques used in production, not a toy with no connection to reality. Meta's fastText project distributes a language-identification model trained to distinguish over a hundred languages, using frequency counts of short character sequences (rather than whole words) learned from very large text collections — the same core idea of "count patterns per class from labelled data, then compare a new example's pattern against each class's learned pattern" scaled up with far more data and finer-grained features. Google Chrome's built-in translate-detection feature (CLD3) works on a related principle. The reason those systems use character sequences instead of whole words is exactly the weakness you can already see in this chapter's model: a word-based model draws a total blank on any word it has never seen, while short character sequences like "kya", "aai", or "tha" keep showing up inside many different Hindi words even when the whole word itself is new, giving the model far more to work with on unfamiliar input.

Check Your Understanding

• Trace detect_language by hand on the input "கல் mera exam hai" — a sentence mixing one Tamil word with Latin ones. Which branch of detect_script does it fall into, and why does max(counts, key=counts.get) decide the outcome even though the sentence contains a Tamil character at all? (Hint: count each script's characters first — Tamil characters here are outnumbered by Latin letters.)
• Compute score("woh accha exam tha", hinglish_counts) by hand, word by word, using the counts given in this chapter (remember tha never appeared in training). What score do you get, and would classify_latin_text return "hinglish" or "unknown"?
• Explain in your own words why classify_latin_text_BUGGY is wrong on tied zero scores, but would give the identical, correct answer to classify_latin_text on any input where e_score and h_score are unequal.
• The training data used only 5 sentences per class. If you doubled it to 10 realistic sentences per class, would you expect the "Cats sleep on soft blankets" problem to disappear completely, shrink, or stay exactly the same? Justify your answer using what out-of-vocabulary words mean for this model.
• Design one new Hinglish test sentence and one new English test sentence, each using at least one word not present anywhere in this chapter's training data. Predict, and then verify by hand, what classify_latin_text outputs for each, and state whether your prediction reveals a limitation of the model.

Summary

Detecting the language of a short piece of Indian text cannot be solved by looking at the alphabet alone, because Hindi typed in Roman script (Hinglish) and English share the exact same character set. The fix is a two-stage pipeline: a fast, reliable Unicode-range check handles native scripts like Devanagari and Tamil outright, and falls through to a statistical word-frequency classifier only for the genuinely ambiguous Latin-script case. That classifier is trained by counting how often each word appears across labelled example sentences for each class, and it scores new text by summing up those learned counts word by word — the same underlying idea, at a smaller scale, behind real language-identification systems like fastText's. Two lessons matter beyond the code itself: first, a score of zero is an absence of evidence, not evidence of the opposite class, and a correct classifier must be able to say "unknown" rather than guess; second, a test accuracy of 100% only means something when the test data honestly represents the unpredictable text the model will meet in the real world, and a small, closed-vocabulary test set can flatter a model that is actually fragile. Together, the training sentences, the scoring function, the tie-handling fix, and the confusion matrix walk through every stage of the AI Project Cycle — problem scoping, data acquisition, modelling, and evaluation — on a problem every Indian smartphone user runs into daily.

Think About It

Think about this: How would you explain ai capstone project: indian language detector to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where ai capstone project: indian language detector is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting ai capstone project: indian language detector to at least 3 other topics you have studied.