Introduction to Attention Mechanisms

Read this sentence once, carefully: "The tiffin didn't fit in the bag because it was too big." Now answer one question before reading on: what does the word it refer to — the tiffin, or the bag?

Almost every student answers "the tiffin," instantly and without effort. But notice what your brain just did. The word "it" sits right next to "was too big," yet to figure out what "it" means, you didn't just look at the words next to it — you reached backward across the whole sentence, compared "tiffin" and "bag," and used the meaning of the word "big" to decide which of the two objects makes sense as the thing that's too big to fit. You gave "it" a direct, weighted link to two words far away, and you gave those two words very different amounts of attention: a lot to "tiffin," almost none to "the," "didn't," or "fit."

That instinctive skill — deciding, for every word, exactly which other words matter and by how much — is precisely what an attention mechanism teaches a neural network to do. This chapter builds that idea from scratch: why older neural networks for language could not do this well, what an attention mechanism computes, and how to trace the arithmetic by hand for a real (if tiny) example.

The problem: one word can depend on a word far away

Sentences like the tiffin-and-bag example are called Winograd-style sentences, named after the type of test the AI researcher Terry Winograd proposed. Change one word — say "big" to "small" — and the answer flips: "The tiffin didn't fit in the bag because it was too small" now makes "it" refer to the bag. No amount of looking at word order or nearby words alone can solve this. You need to compare "it" against every candidate noun in the sentence and combine that comparison with the adjective at the very end.

This kind of long-distance dependency is everywhere in language, not just in trick sentences. Consider a WhatsApp forward that rambles for three paragraphs before finally getting to "and that is why she cancelled the trip" — to know who "she" is, you must connect back to a name mentioned at the very start. Or consider translating a sentence from English to Hindi: English usually puts the verb in the middle of the sentence ("Ravi gave the book to Meera"), while Hindi typically puts the verb at the end ("Ravi ne Meera ko kitab di"). A machine translating this sentence has to hold onto "gave" mentally while it produces several other words first, then place it correctly near the end. The information needed at one position in the sentence often lives at a completely different position.

Why the older approach struggled

Before attention mechanisms became standard (around 2014-2015), the leading approach to tasks like machine translation was a design called sequence-to-sequence: one part of the network, the encoder, read the input sentence one word at a time and squeezed everything it had read into a single fixed-size list of numbers — a "context vector." A second part, the decoder, then had to generate the entire output sentence using only that one compressed vector.

Think of it like being asked to summarize an entire chapter of your history textbook into exactly 20 words, and then being quizzed on chapter details using only your 20-word summary — no going back to the book. For a short sentence this works fine. But as sentences get longer, more and more information has to be squeezed into the same fixed-size vector, and early details start getting overwritten or blurred by later ones, the same way your 20-word summary would lose the fine details of page 3 by the time you finished summarizing page 30. Researchers observed exactly this: translation quality dropped sharply as sentences got longer, because the single context vector became an information bottleneck.

In 2014, researchers Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed a fix: instead of forcing the decoder to work from one compressed summary, let it look back at every word of the input sentence at every step, and learn how much weight to give each one. This was the first widely used attention mechanism. A few years later, in 2017, a Google research team (Vaswani et al., in the paper titled "Attention Is All You Need") went further and built a network — the Transformer — that used attention as its main building block and removed the step-by-step reading of RNNs entirely. Transformers now power the large language models behind most modern AI text and translation tools.

The core idea: search instead of squeeze

Here is the central shift in thinking. Instead of compressing the whole sentence into one vector and hoping nothing important gets lost, attention lets every word directly ask a question of every other word: "How relevant are you to me, right now?" It then builds its understanding of that word as a weighted mixture of the other words, where the weights come from those relevance scores.

This is exactly how a library catalog search works, and that analogy gives us the three pieces every attention mechanism is built from:

• Query — what you are looking for. When "it" is trying to figure out what it refers to, "it" issues a query: "I'm looking for a physical object that something else was compared to."
• Key — the label attached to each item you could retrieve, used only for matching against the query. Every word in the sentence carries a key: "tiffin" has a key that roughly says "I am a physical, packable object." "bag" has a similar key. "because" has a key that says almost nothing relevant.
• Value — the actual content you get back once a match is found. Once "it" decides "tiffin" is the best match, it doesn't just get back the label "tiffin" — it pulls in "tiffin"'s value, the rich content describing what a tiffin actually is, and blends that into its own understanding.

In a library, you type a query into the search box, the system compares it against the keys (titles, subjects, authors) of every book, ranks the books by how well they match, and hands you back the values (the actual books) — mostly the top matches, but technically a little bit of everything is "returned," just with vanishingly small relevance for the bad matches. Attention does exactly this, except it returns a mathematical blend of all the values, weighted by match quality, rather than a single best book.

Turning "how relevant" into a number: the dot product

To make "relevance" computable, every word is first represented as a list of numbers — an embedding — where similar meanings tend to produce similar number patterns. Real systems use embeddings with hundreds of numbers per word; to trace every step by hand, we'll use toy embeddings with just 2 numbers per word.

The simplest way to measure how well two number-lists "match" is the dot product: multiply the numbers in matching positions, then add up the results. If Query = (1, 1) and a Key = (2, 1), the dot product is (1×2) + (1×1) = 2 + 1 = 3. A bigger dot product means the two vectors point in a more similar direction — a stronger match. This single arithmetic operation, repeated once per word, is the entire "search" step of attention.

A full worked example, traced by hand

Let's compute real attention weights for our tiffin sentence. We'll compute them from the point of view of the word "it" — that is, "it" is the word issuing the query, and every word (including "it" itself) offers a key and a value. These numbers are simplified teaching vectors, not numbers from a trained model, chosen so the arithmetic stays clean while still landing on a sensible answer.

Query vector for "it": (1, 1). (In a real Transformer, Query, Key, and Value vectors are each produced by multiplying the word's embedding by three separate learned weight matrices, called W_Q, W_K, and W_V. We're skipping that multiplication step here and simply giving you the resulting numbers directly, so we can focus on what happens after Q, K, and V exist.)

Step 1 — Raw scores. Take the dot product of the query with every key:

• tiffin: (1×2) + (1×1) = 3
• bag: (1×0) + (1×1) = 1
• it: (1×1) + (1×1) = 2
• big: (1×1) + (1×0) = 1

Step 2 — Turn scores into weights (softmax). We now have four numbers: 3, 1, 2, 1. We want to turn them into "how much attention to pay," expressed as percentages that add up to 100%. The obvious idea — just divide each score by the total — has a real problem: dot products can come out negative when two vectors point in opposite directions, and a negative percentage of attention makes no sense.

The fix used in every attention mechanism is called softmax. First, raise the special number e (approximately 2.718) to the power of each score — this instantly makes every value positive, no matter how negative the original score was, and it exaggerates the gap between the winning score and the rest (a score of 3 doesn't just beat a score of 1 by a little; e³ beats e¹ by a lot). Then divide each result by the sum of all of them, so the final numbers add up to exactly 1.

Using a calculator (e¹ ≈ 2.718, e² ≈ 7.389, e³ ≈ 20.086):

• e³ = 20.086 (tiffin)
• e² = 7.389 (it)
• e¹ = 2.718 (bag)
• e¹ = 2.718 (big)

Total = 20.086 + 7.389 + 2.718 + 2.718 = 32.911

Weights: tiffin = 20.086 ÷ 32.911 ≈ 61.0%, it = 7.389 ÷ 32.911 ≈ 22.5%, bag = 2.718 ÷ 32.911 ≈ 8.3%, big = 2.718 ÷ 32.911 ≈ 8.3% (adds to 100.1% only because of rounding).

(A technical note for the record: real Transformers divide the raw scores by the square root of the key vector's length — here that would be √2 ≈ 1.41 — before applying softmax, to stop the scores from growing huge when vectors have hundreds of numbers instead of 2. This is called scaled dot-product attention. We've skipped that division since our 2-number vectors never get large enough to cause trouble, but you should know professional systems include it.)

Step 3 — Blend the values. Multiply each word's Value vector by its weight, and add the results together:

• 0.610 × (5, 2) = (3.05, 1.22)
• 0.225 × (2, 2) = (0.45, 0.45)
• 0.083 × (5, 6) = (0.41, 0.50)
• 0.083 × (0, 8) = (0.00, 0.66)

Adding the x-coordinates: 3.05 + 0.45 + 0.41 + 0.00 = 3.91. Adding the y-coordinates: 1.22 + 0.45 + 0.50 + 0.66 = 2.83.

The new, attention-updated vector for "it" is approximately (3.91, 2.83). Compare that to "tiffin"'s original value vector, (5, 2) — the new "it" vector sits much closer to "tiffin" than to "bag" (5, 6) or "big" (0, 8). This is the entire point: after passing through attention, the representation of "it" is no longer a generic, ambiguous . It now carries mostly tiffin's meaning mixed with a little of its own and small traces of "bag" and "big" — because that's where the sentence told it to look.

Here is the same three steps as runnable Python, using only plain lists and the standard math library — no special packages required:

import math

# Toy word vectors (real models use hundreds of numbers per word;
# we use 2 numbers per word so every step can be traced by hand)
K = {
    "tiffin": (2, 1),
    "bag":    (0, 1),
    "it":     (1, 1),
    "big":    (1, 0),
}
V = {
    "tiffin": (5, 2),
    "bag":    (5, 6),
    "it":     (2, 2),
    "big":    (0, 8),
}

query = (1, 1)   # Query vector for the word "it"

def dot(a, b):
    return a[0]*b[0] + a[1]*b[1]

# Step 1: raw attention scores
scores = {word: dot(query, key) for word, key in K.items()}
print(scores)
# {'tiffin': 3, 'bag': 1, 'it': 2, 'big': 1}

# Step 2: turn scores into weights that add up to 1 (softmax)
exp_scores = {word: math.exp(s) for word, s in scores.items()}
total = sum(exp_scores.values())
weights = {word: e / total for word, e in exp_scores.items()}
print({word: round(w, 3) for word, w in weights.items()})
# {'tiffin': 0.61, 'bag': 0.083, 'it': 0.225, 'big': 0.083}

# Step 3: blend the Value vectors using these weights
output = [0, 0]
for word, w in weights.items():
    output[0] += w * V[word][0]
    output[1] += w * V[word][1]

print([round(x, 2) for x in output])
# [3.91, 2.83]

Run this in your head or on a computer and confirm each printed line matches the comment beneath it — that is exactly the by-hand arithmetic above, just expressed as code.

Seeing it as a diagram

From one word to a whole sentence: self-attention

We only computed one row of a much bigger picture. In an actual Transformer, every word in the sentence plays the role of "it" once: "tiffin" issues its own query and computes its own weights over every other word, "bag" does the same, and so on. Because the queries are being compared against keys drawn from the same sentence, this specific setup is called self-attention — the sentence is attending to itself. Stack these rows together and you get a full attention matrix: one row per word, one column per word, where row i, column j tells you how much word i attends to word j. Every row is computed with the same three steps we just traced by hand — dot product, softmax, weighted blend of values — just with a different query vector each time.

(There's also cross-attention, used in translation: the queries come from the sentence being generated in Hindi, say, while the keys and values come from the original English sentence. The mechanics — dot product, softmax, weighted blend — are identical; only the source of the queries changes.)

This row-by-row structure is also why Transformers train faster than the RNNs they replaced. An RNN has to process word 1, then word 2, then word 3, in strict order, because each step depends on the output of the previous one. Every row of a self-attention matrix, by contrast, can be computed independently and simultaneously, since none of them depend on each other — which means a graphics processing unit (GPU) can compute all the rows in parallel instead of waiting through hundreds of sequential steps.

Two misconceptions worth correcting now

Misconception 1: "Attention weights tell us which words are objectively 'the most important' in a sentence." They don't — not in any absolute sense. Attention weights are numbers a network learns during training, adjusted over and over by gradient descent so that the network's final predictions get closer to the correct answer. They happened to align with human intuition in our tiffin example because we deliberately chose toy numbers that would. A real trained network's weights reflect whatever pattern helped it minimize error on its training data — which usually correlates with human judgments of relevance, but is a learned statistical pattern, not a hand-coded rule about "importance."

Misconception 2: "Since every word can directly attend to every other word, attention already knows the order of the sentence." This is false, and it's a genuinely important limitation. Look again at Step 1 of our worked example: the dot product between "it" and "tiffin" would come out exactly the same number whether "tiffin" was the 2nd word of the sentence or the 20th — the calculation only uses the two vectors' numbers, never their positions. On its own, self-attention is permutation-invariant: shuffle the words of the input, and the attention scores for any given pair just get shuffled along with them, with no penalty for word order being scrambled. This is exactly why real Transformers add something called a positional encoding — extra numbers baked into each word's embedding before attention even runs, specifically so the network can tell "Ravi hit the ball" apart from "the ball hit Ravi."

Where this shows up outside translation

Attention mechanisms were first popularized in machine translation, but the same three-step recipe — query, key, value, softmax, blend — now appears far beyond language. Vision Transformers apply self-attention across patches of an image instead of words in a sentence, letting a network relate a cat's ear in one corner of a photo to its tail in another corner directly, without scanning pixel-by-pixel in between. DeepMind's AlphaFold2, which predicts the 3-D shape a protein folds into from its chain of amino acids, also relies on attention layers to let distant parts of the amino acid chain influence each other, since amino acids that are far apart in the sequence can end up physically close once the protein folds. Closer to home, AI4Bharat, a research group at IIT Madras, built IndicTrans — an open translation system for Indian languages — on top of exactly this Transformer, attention-based architecture, adapted for the specific challenge of translating between languages like Hindi, Tamil, Bengali, and English.

For your CBSE Computer Science / Informatics Practices exam, remember the layered picture: a neural network is the broad family; a Transformer is one particular network architecture within that family; and attention (specifically self-attention and cross-attention) is the specific computational mechanism — query, key, value, dot product, softmax, weighted sum — that Transformers are built out of. Getting that hierarchy straight, and being able to name what problem attention actually solves (the fixed-size context bottleneck of older sequence models), is worth more marks than memorizing the word "Transformer" alone.

Check your understanding

1. Given Query = (2, 0), Key_a = (1, 1), and Key_b = (0, 3), compute the raw attention score (dot product) between the query and each key.
2. Why do attention mechanisms exponentiate the raw scores (using e^x) before normalizing them into weights, instead of simply dividing each score by the sum of all the scores?
3. In our worked example, "it" ended up paying only 8.3% attention to "bag" even though "bag" is a perfectly reasonable physical object. What determined that low number — was it a rule someone programmed in, or something else?
4. True or false, with a reason: "A self-attention layer, by itself, can tell that 'the dog bit the man' and 'the man bit the dog' describe different events."
5. Name the three vectors that get computed for every word before an attention score can be calculated.

Answers to check yourself: (1) Score with Key_a = (2×1)+(0×1) = 2; score with Key_b = (2×0)+(0×3) = 0. (2) Raw dot-product scores can be negative, and a "negative percentage of attention" is meaningless; exponentiating makes every value positive first, and it also sharpens the gap so the network can commit more strongly to its best match rather than spreading attention almost evenly. (3) It was neither a hand-written rule nor an accident of "reasonableness" — in our toy example it came from the specific numbers we chose for the Key vectors; in a real trained network it would come from weights learned during training on huge amounts of text, adjusted to reduce prediction error. (4) False — self-attention's dot products only depend on the word vectors involved, not their positions, so on their own the two sentences would score identically; a Transformer only tells them apart because of the added positional encodings. (5) Query, Key, and Value.

Summary

• Older sequence models compressed an entire input sentence into one fixed-size vector, creating an information bottleneck that got worse as sentences grew longer.
• Attention mechanisms fix this by letting every word directly compare itself against every other word and combine their information, weighted by relevance, instead of relying on one compressed summary.
• Every word produces three vectors: a Query (what it's looking for), a Key (what it advertises about itself for matching), and a Value (the content it actually contributes).
• Attention scores are computed as dot products between a Query and each Key, then converted into weights that sum to 1 using softmax (exponentiate, then normalize) — this is called scaled dot-product attention when the scores are also divided by the square root of the key vector's length.
• The final output for a word is the weighted sum of every word's Value vector, using those softmax weights — this is exactly what we computed by hand, landing on (3.91, 2.83) for "it," pulled toward "tiffin."
• Self-attention compares a sentence against itself; cross-attention compares one sequence's queries against a different sequence's keys and values (as in translation).
• Self-attention has no built-in sense of word order — that's added separately through positional encodings — and its weights are learned statistical patterns from training data, not hand-coded importance rules.
• Attention is the core building block of the Transformer architecture (2017), which now underlies most modern language and translation systems, including India's own IndicTrans project from AI4Bharat.

Think About It

Think about this: How would you explain introduction to attention mechanisms to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.