Word Embeddings — Turning Language into Mathematics

Type "sasta phone under 10000" into a shopping app, and it happily shows you results titled "affordable smartphone under ₹10,000" — even though not one word in your query matches the listing's title exactly. "Sasta" isn't "affordable." "Phone" isn't "smartphone." Yet the app knows they mean almost the same thing. How does a machine that only understands numbers know that two completely different strings of letters point at the same idea? The answer is a technique called word embeddings — a way of converting every word in a language into a list of numbers so carefully chosen that words with similar meanings end up as numbers that are close together, and words with different meanings end up far apart. This chapter builds that idea from scratch, with real numbers you can compute by hand.

Step 1: The obvious approach, and why it fails

Computers do arithmetic on numbers, not letters, so before any machine learning model can process text, every word must first become a number. The most naive idea is to just number the words: assign "aam" the number 1, "apple" the number 2, "banana" the number 3, "cricket" the number 47, and so on, the way words are ordered in a dictionary. This fails immediately and badly. If "apple" is 2 and "cricket" is 47, does that mean cricket is roughly 23 times more "something" than apple? Is "banana" (3) closer in meaning to "apple" (2) just because 3 is closer to 2 than 47 is? Of course not — the numbering is arbitrary. Swap the dictionary order and every relationship the model might have "learned" from those numbers becomes meaningless. A representation where the actual values carry no real information about meaning is not usable for language tasks.

Step 2: One-hot vectors — numbers with no meaning, but at least no false meaning

The standard fix for arbitrary numbering is called one-hot encoding. Instead of one arbitrary number, every word gets its own vector — a list of numbers — as long as the entire vocabulary, filled with zeros except for a single 1 at that word's position. If your vocabulary is ["mango", "apple", "cricket", "football"], then "mango" becomes [1, 0, 0, 0] and "cricket" becomes [0, 0, 1, 0]. No word is arbitrarily "closer" to another anymore, because every pair of distinct words is exactly the same distance apart. Let's check that in code.

vocab = ["mango", "apple", "cricket", "football"]

def one_hot(word):
    vec = [0] * len(vocab)
    vec[vocab.index(word)] = 1
    return vec

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

print(one_hot("mango"))                        # [1, 0, 0, 0]
print(one_hot("apple"))                         # [0, 1, 0, 0]
print(dot(one_hot("mango"), one_hot("apple")))  # 0
print(dot(one_hot("mango"), one_hot("cricket"))) # 0

Trace it: one_hot("mango") finds "mango" at index 0 in vocab, so it builds [1,0,0,0]. one_hot("apple") finds "apple" at index 1, giving [0,1,0,0]. The dot function multiplies matching positions and adds them up: position 0 gives 1×0, position 1 gives 0×1, positions 2 and 3 give 0×0 — every term is zero, so the total is 0. The same is true for any two distinct one-hot vectors, no matter how related the words actually are. Mango and apple — both sweet fruits you'd find at a Nashik fruit stall — get exactly the same "closeness score" (zero) as mango and cricket, a fruit and a sport that have nothing in common. One-hot vectors solve the false-similarity problem, but they throw away real similarity too. They also don't scale: a vocabulary of 50,000 words means every single word is a mostly-empty list of 50,000 numbers. We need something better — vectors that are both small and meaningful. That is what a word embedding is: a short, dense list of numbers (typically somewhere between 50 and 300 of them, not tens of thousands) where the actual values encode something real about how the word is used.

Step 3: A word is known by the company it keeps

The linguist J.R. Firth put it in one sentence: "You shall know a word by the company it keeps." Words that tend to appear in similar surrounding contexts tend to have similar meanings. "Mango" and "apple" both show up near words like "sweet," "eat," and "ripe." "Cricket" and "football" both show up near words like "bat," "ball," and "goal" — well, football uses "ball" too, but not "bat." This idea is called the distributional hypothesis, and it is the foundation on which every modern word embedding is built. Instead of trying to hand-write what a word "means," we let the model learn meaning purely from statistics: which words tend to occur near which other words, across huge amounts of real text.

Let's build a tiny embedding by hand using nothing but counting, so the mechanism is completely transparent before we let a computer take over. Here is a seven-sentence toy corpus:

1. I eat a sweet mango every morning.
2. She bought a ripe mango and a red apple from the market.
3. The apple was sweet and crunchy.
4. He plays cricket with a bat every evening.
5. Cricket needs a bat and a ball.
6. Football needs a ball and a goal.
7. She scored a goal playing football.

Pick six context words to track: sweet, ripe, eat, bat, ball, goal. For each target word, count how many of the sentences containing it also contain each context word. That count becomes one coordinate of the target word's vector, in the fixed order [sweet, ripe, eat, bat, ball, goal]:

• mango appears in sentence 1 (contains "sweet" and "eat") and sentence 2 (contains "ripe") → [1, 1, 1, 0, 0, 0]
• apple appears in sentence 2 (contains "ripe") and sentence 3 (contains "sweet") → [1, 1, 0, 0, 0, 0]
• cricket appears in sentence 4 (contains "bat") and sentence 5 (contains "bat" and "ball") → [0, 0, 0, 2, 1, 0]
• football appears in sentence 6 (contains "ball" and "goal") and sentence 7 (contains "goal") → [0, 0, 0, 0, 1, 2]

Notice what just happened: mango and apple now have almost identical vectors, purely because they tend to appear near the same context words — nobody told the computer that both are fruits. That is the entire trick of embeddings, done by hand with a seven-sentence corpus.

Step 4: Measuring closeness — cosine similarity

To turn "the vectors look similar" into a precise number, we use cosine similarity. It measures the angle between two vectors, ignoring their length, and produces a score between −1 and 1: 1 means the vectors point in exactly the same direction (maximally similar), 0 means they are perpendicular (unrelated), and −1 means they point in opposite directions. The formula is:

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

where A · B is the dot product (multiply matching positions, add them up — the same dot function from Step 2) and |A| is the vector's magnitude, computed as √(sum of each coordinate squared). Let's compute it for our fruit and sport vectors.

import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(y * y for y in b))
    return dot / (mag_a * mag_b)

# order: [sweet, ripe, eat, bat, ball, goal]
mango    = [1, 1, 1, 0, 0, 0]
apple    = [1, 1, 0, 0, 0, 0]
cricket  = [0, 0, 0, 2, 1, 0]
football = [0, 0, 0, 0, 1, 2]

print(round(cosine_similarity(mango, apple), 2))     # 0.82
print(round(cosine_similarity(mango, cricket), 2))   # 0.0
print(round(cosine_similarity(cricket, football), 2)) # 0.2

Trace the first call. dot = (1×1) + (1×1) + (1×0) + (0×0) + (0×0) + (0×0) = 1 + 1 + 0 = 2. mag_a = √(1² + 1² + 1² + 0 + 0 + 0) = √3 ≈ 1.732. mag_b = √(1² + 1² + 0 + 0 + 0 + 0) = √2 ≈ 1.414. So the result is 2 ÷ (1.732 × 1.414) = 2 ÷ 2.449 ≈ 0.8165, which rounds to 0.82 — strongly similar, exactly as we'd hope for two fruits. For mango and cricket, every position where one vector is non-zero, the other is zero, so every product in the dot product is zero, the dot product is 0, and the similarity is 0.0 — completely unrelated, even though both are common Indian nouns of similar "importance." Cricket and football score only 0.2, lower than you might expect for two sports — because our seven-sentence toy corpus barely gives them any shared context beyond the word "ball." This is a genuine and important limitation to notice: with more real text (imagine millions of sentences instead of seven), cricket and football would keep co-occurring with words like "team," "player," "stadium," and "match," and their similarity score would climb much higher. Small corpora give noisy, weak embeddings; this is precisely why real embeddings are trained on enormous amounts of text — billions of words scraped from books, articles, and web pages — rather than a handful of hand-picked sentences.

Step 5: From counting to learning

Counting co-occurrences by hand, as we just did, is actually how one of the earliest successful embedding methods worked. But it has a practical problem: for a real vocabulary of even 50,000 words, a full co-occurrence table would need 50,000 × 50,000 = 2.5 billion entries, almost all of them zero. Modern embedding algorithms — the most famous being word2vec, introduced by Tomáš Mikolov and colleagues at Google in 2013, and GloVe, from Stanford in 2014 — replace explicit counting with a small neural network trained on a simple prediction task, so the model never has to build that giant table.

Word2vec has two common setups. In the skip-gram version, the network slides a small window across the text (say, two words on either side) and is trained to predict the surrounding context words given the current target word. In the CBOW (continuous bag-of-words) version, it does the reverse: predict the target word from its surrounding context. Either way, the network never explicitly cares about getting the prediction perfectly right — what we actually want is a side effect of the training process. Inside the network is a hidden layer of, say, 200 numbers per word. To make good predictions across millions of sentences, the network is forced to adjust those 200 numbers so that words used in similar contexts end up with similar internal representations — because that's the only way to predict their shared context words accurately. Once training finishes, we throw away the prediction task entirely and keep just those 200 numbers per word. That is the embedding: a dense vector, learned as a byproduct of a prediction task, not counted by hand and not designed by a human.

Step 6: Vector arithmetic — king − man + woman ≈ queen

The most striking property of trained embeddings is that meaningful relationships between words show up as consistent directions in the vector space, and you can do algebra with them. The famous illustration: take the embedding vector for "king," subtract the vector for "man," add the vector for "woman," and the resulting vector lands almost exactly on the embedding for "queen." In plain language: the direction you move to go from "man" to "king" (roughly, "add royalty") is nearly the same direction you'd move to go from "woman" to "queen."

Real word2vec vectors have 100–300 dimensions, which we can't draw or hand-compute here, so let's build a simplified 2-dimensional toy version to see exactly why the arithmetic works, using coordinates on two made-up axes — how "royal" a word is, and how strongly "female" it is:

man   = (1, 0)
woman = (1, 5)
king  = (8, 0)
queen = (8, 5)

result = (king[0] - man[0] + woman[0],
          king[1] - man[1] + woman[1])

print(result)            # (8, 5)
print(result == queen)   # True

Trace it: the first coordinate is 8 − 1 + 1 = 8. The second coordinate is 0 − 0 + 5 = 5. So result = (8, 5), which is exactly the coordinates we assigned to "queen," so result == queen prints True. This works because "king" and "man" differ only in the royalty coordinate (both have 0 in the gender coordinate), and "woman" simply adds the female coordinate back in without touching royalty — subtracting "man" cancels the shared "male, commoner" component and adding "woman" reintroduces exactly the "female" component while keeping "royal." The diagram below shows this as a picture: the arrow from "man" to "king" (add royalty) is parallel to and the same length as the arrow from "woman" to "queen," and the arrow from "man" to "woman" (add female) is parallel to and the same length as the arrow from "king" to "queen."

Common misconception: embeddings do not have human-readable dimensions

The toy example above is genuinely useful for building intuition, but it can plant a false idea, so let's correct it directly: a real trained embedding does not have one coordinate that means "royalty" and another that means "gender." In our 2-dimensional toy, we chose the axes to mean something, in order to make the arithmetic easy to follow. In an actual word2vec or GloVe model, each of the 100–300 coordinates is just a number that the training process happened to land on, and no individual coordinate has any interpretable meaning in isolation — coordinate number 47 is not "royalty" and coordinate 112 is not "gender." What does hold up in real trained embeddings is that certain directions — patterns that cut diagonally across many coordinates at once — correspond to consistent relationships like gender or verb tense, and researchers discover these directions by analyzing the vectors after training, not by designing them in beforehand. The king/queen analogy is a real, well-documented result, but it is a property that emerges from training on huge amounts of text, not a labeled feature built into the model.

A second common misconception is worth naming here too: word embeddings do not store a dictionary definition of a word anywhere inside them. There is no coordinate that encodes "a large striped wild cat" for the word "tiger." All that exists is a list of numbers shaped entirely by which other words tend to appear nearby across the training text. This also explains a genuine weakness of classic word embeddings: a word like "bank" gets exactly one vector, which has to awkwardly average together "river bank" and "savings bank" contexts, because word2vec and GloVe assign a single fixed vector per word regardless of which meaning is intended in a given sentence. This limitation is precisely what motivated later, more advanced contextual models, which produce a different vector for the same word depending on the sentence it appears in — a topic that builds directly on the foundations in this chapter.

Why this matters

Once words are numbers that respect meaning, a huge range of language technology becomes possible with fairly simple mathematics on top. A search engine can match your query to relevant results even when the exact words differ, because it compares embedding vectors instead of exact text — this is exactly what let "sasta phone" find "affordable smartphone" in the opening example. Spam filters and sentiment classifiers can generalize from words they've seen in training to related words they haven't, because similar words land near each other in the vector space. Machine translation systems, including the ones behind everyday Hindi–English translation apps, rely on embeddings as their starting representation of each word before any translation logic runs. None of this works if words are represented as arbitrary indices or as one-hot vectors with no notion of closeness — embeddings are the layer that turns raw text into something a mathematical model can meaningfully reason about.

Check your understanding

1. Two vectors A = [2, 0, 1] and B = [1, 0, 2] represent two words based on context-word counts. Compute the dot product, the magnitude of each vector, and the cosine similarity, showing every step.
2. Explain in your own words why the dot product of any two different one-hot vectors is always exactly 0, using the definition of the dot product.
3. A classmate says: "Word embeddings work by looking up each word's dictionary meaning and converting it into numbers." Explain precisely what is wrong with this statement and what embeddings actually use instead.
4. Using the toy coordinates from this chapter — man (1,0), woman (1,5), king (8,0) — suppose "prince" is at (5,0). Using the same royal-direction and female-direction logic, predict a plausible coordinate for "princess" and justify it with the vector arithmetic pattern.
5. Our toy corpus gave cricket and football a cosine similarity of only 0.2, lower than intuition suggests two sports should be. Explain, in terms of corpus size and co-occurrence counts, why this number is artificially low and what would fix it.

Summary

• Computers require every word to become a number before any model can process it; arbitrary dictionary-order numbering creates false relationships and must be avoided.
• One-hot vectors fix the false-relationship problem but make every pair of distinct words equally "unrelated" (cosine similarity always 0) and scale terribly with vocabulary size.
• The distributional hypothesis — a word is known by the words that surround it — is the foundation of embeddings: similar contexts produce similar vectors, which we verified by hand-counting co-occurrences in a small corpus and computing cosine similarity (mango–apple ≈ 0.82, mango–cricket = 0.0).
• Cosine similarity, (A·B)/(|A||B|), measures the angle between two vectors and is the standard way to score how similar two embeddings are, independent of their length.
• Real embeddings such as word2vec (skip-gram/CBOW) and GloVe are learned by training a neural network on a word-prediction task; the embedding is the trained internal representation left over once the prediction task itself is discarded.
• Meaningful relationships appear as directions in the vector space, letting analogies like king − man + woman ≈ queen work through simple vector arithmetic.
• Individual coordinates in a real trained embedding are not human-labeled concepts like "royalty" or "gender" — only combinations of many coordinates form interpretable directions, and even those are discovered after training, not designed in. Classic embeddings also assign only one fixed vector per word, which is a genuine limitation for words with multiple meanings.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where word embeddings — turning language into mathematics is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting word embeddings — turning language into mathematics to at least 3 other topics you have studied.