Positional Encoding: Teaching Order

"Ravi beats Meera in the chess final" and "Meera beats Ravi in the chess final" contain exactly the same three content words — Ravi, Meera, beats — arranged differently. Swap the order and the winner swaps too. Word order is not decoration on top of meaning; for a huge number of sentences, it is the meaning. So here is an uncomfortable question about the self-attention mechanism you have already studied: does it actually notice order at all, or does it just look at which words are present?

Let's check with real numbers, not intuition.

A concrete proof that plain self-attention is blind to order

Take a toy sentence with two tokens, "Ravi" and "Meera", and give each a simple embedding vector (2-dimensional, just for hand computation): x_Ravi = [2, 0] and x_Meera = [0, 2]. To keep the arithmetic transparent, use the simplest possible self-attention — no learned weight matrices, so query = key = value = the embedding itself. Recall the attention formula from your earlier chapter:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d)) V

Stack the two embeddings as rows of a matrix X = [[2,0],[0,2]], so Q = K = V = X. First compute the raw similarity scores X X^T: each entry is a dot product between two token vectors.

X X^T = [[x_Ravi . x_Ravi,  x_Ravi . x_Meera ],
         [x_Meera. x_Ravi,  x_Meera. x_Meera]]
      = [[4, 0],
         [0, 4]]

Scale by 1/sqrt(d) = 1/sqrt(2) ≈ 0.7071: the scaled score matrix is [[2.828, 0], [0, 2.828]]. Apply softmax row by row. For row 1 (Ravi's row): e^2.828 ≈ 16.9, e^0 = 1, so the row becomes [16.9/17.9, 1/17.9] ≈ [0.944, 0.056]. By the identical calculation, row 2 (Meera's row) becomes [0.056, 0.944].

Now multiply by V = X to get the final output for each token:

out_Ravi  = 0.944 * [2,0] + 0.056 * [0,2] = [1.888, 0.112]
out_Meera = 0.056 * [2,0] + 0.944 * [0,2] = [0.112, 1.888]

Now rerun the exact same computation with the sentence reversed: Meera first, Ravi second, so X' = [[0,2],[2,0]]. Compute X' X'^T: row 1 is now Meera's row, and x_Meera . x_Meera = 4, x_Meera . x_Ravi = 0 — the matrix is still [[4,0],[0,4]], because dot products don't care which token you list first. Softmax gives the same [0.944, 0.056] and [0.056, 0.944] rows. Multiplying by V' = [x_Meera, x_Ravi]:

out_Meera(now row 1) = 0.944*[0,2] + 0.056*[2,0] = [0.112, 1.888]
out_Ravi (now row 2) = 0.056*[0,2] + 0.944*[2,0] = [1.888, 0.112]

Look closely: Ravi's output vector is [1.888, 0.112] whether Ravi is the first word or the second word in the sentence. Meera's output vector is [0.112, 1.888] regardless of where she sits. Reversing the sentence didn't change either token's computed representation at all — it only relabeled which row holds which token's already-fixed answer. In general, for any permutation matrix P that reorders the input rows, Attention(PX) = P · Attention(X): the outputs get shuffled in exactly the order the inputs were shuffled, but no output vector's value changes. This property is called permutation equivariance, and it is a direct, provable consequence of the fact that every step in the formula — dot products, softmax, weighted sums — treats the tokens as an unordered set of vectors. Self-attention, exactly as we've built it so far, has no way to tell "word 1" from "word 5." It processes a sequence the way a bag processes marbles: contents matter, arrangement doesn't.

This is a real architectural gap, not a minor detail. A recurrent network (RNN) reads tokens one at a time and folds each one into a hidden state before seeing the next, so order is baked in by the mechanics of the loop itself — that's precisely the property transformers gave up in exchange for processing all tokens in parallel (the main reason transformers train faster than RNNs on modern hardware). Something has to put position information back in, deliberately, before attention is applied. That something is positional encoding: a vector added to each token's embedding that depends only on its position in the sequence, so that the network's very first layer already knows position 3 is different from position 7 — the attention mechanism itself never has to.

Two tempting fixes that fail

Misconception: "just tack the index on." A natural first idea is to append the raw position number to each embedding, so token 0 gets a bonus feature of 0, token 1 gets 1, token 500 gets 500. This fails for three concrete reasons. First, magnitude: transformer embeddings are typically initialised and normalised to have entries of order 1. A raw index of 500 would swamp every other feature — the network would see almost nothing but the position, wiping out the word's actual meaning. Second, no useful structure: the network has to learn, purely from data, that "312 is close to 313" in some usable sense, when nothing in the representation makes that closeness explicit or easy to compute with a linear layer. Third, and most seriously, generalisation: if training sentences never exceed 500 tokens, the model has literally never seen the number 501 during training. A brand-new integer at test time is just an unfamiliar input, and there's no guarantee the model handles it sensibly.

Second attempt: normalise position to [0, 1] by dividing by sentence length. This avoids the magnitude problem but introduces a worse one: "the 5th word" means 5/10 = 0.5 in a 10-word sentence but 5/500 = 0.01 in a 500-word document. The same relative gap — one word apart — is stretched or squeezed completely differently depending on how long the sentence happens to be. A fixed step in this normalised coordinate has no consistent meaning across sequences of different lengths, which is exactly the property we need position encoding to have.

The sinusoidal idea: many clock hands ticking at different speeds

Here's the intuition the original Transformer paper (Vaswani et al., "Attention Is All You Need," 2017) used, and it is genuinely elegant. Think of an analogue clock face with three hands — seconds, minutes, hours — each ticking at a different speed. Read off all three positions together and you can pinpoint the exact second within a 12-hour span, even though no single hand alone can: the second hand alone is ambiguous (it repeats every 60 seconds), but second + minute + hour together give a unique, bounded reading.

Positional encoding builds the same idea for a sequence index. Instead of one unbounded number, assign each position a set of coordinates, each oscillating as a sine or cosine wave, but at a different frequency. The fastest-oscillating coordinate changes almost every step (like the second hand — good for telling neighbouring positions apart). The slowest-oscillating coordinate barely moves across the whole sequence (like the hour hand — good for telling far-apart regions of a long sequence apart). Every value stays inside [-1, 1] no matter how large the position gets, because sine and cosine are always bounded — this directly fixes the magnitude problem from the naive-index attempt.

The formula, defined term by term

For a model with embedding dimension d_model (an even number), position pos = 0, 1, 2, ..., and dimension-pair index i = 0, 1, ..., d_model/2 - 1, the original paper defines:

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

Read this as: dimension pair i is a wave with angular frequency w_i = 1 / 10000^(2i/d_model), so its wavelength is λ_i = 2π / w_i = 2π · 10000^(2i/d_model). At i = 0, the wavelength is 2π ≈ 6.28 positions — an extremely fast wave, cycling roughly every 6 tokens. At the largest i (close to d_model/2 - 1), the wavelength approaches 2π · 10000 ≈ 62,832 positions — a wave so slow it looks almost flat across any realistic sentence length. Between these extremes, the wavelengths form a smooth geometric progression, exactly like a bank of clock hands running from "ticks every second" to "ticks every few hours." Every position gets a unique combination of these d_model/2 readings — its own fingerprint — added directly onto its word embedding before the first attention layer runs.

Computing it by hand: a worked example with d_model = 8

Choose d_model = 8 specifically because the arithmetic becomes clean. There are i = 0, 1, 2, 3, and since 10000 = 10^4, the exponent simplifies: 10000^(2i/8) = 10000^(i/4) = 10^(4 · i/4) = 10^i. So the four frequency denominators are exactly 1, 10, 100, 1000:

PE(pos,0)=sin(pos/1)    PE(pos,1)=cos(pos/1)
PE(pos,2)=sin(pos/10)   PE(pos,3)=cos(pos/10)
PE(pos,4)=sin(pos/100)  PE(pos,5)=cos(pos/100)
PE(pos,6)=sin(pos/1000) PE(pos,7)=cos(pos/1000)

All angles are in radians. Plugging in pos = 0, 1, 2, 3 (using sin(1)=0.8415, cos(1)=0.5403, sin(2)=0.9093, cos(2)=-0.4161, sin(3)=0.1411, cos(3)=-0.9900, standard values you can check on any scientific calculator):

pos=0: [0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000]
pos=1: [0.8415, 0.5403, 0.0998, 0.9950, 0.0100, 1.0000, 0.0010, 1.0000]
pos=2: [0.9093,-0.4161, 0.1987, 0.9801, 0.0200, 0.9998, 0.0020, 1.0000]
pos=3: [0.1411,-0.9900, 0.2955, 0.9553, 0.0300, 0.9996, 0.0030, 1.0000]

Notice dimension 0 swings wildly (0 → 0.84 → 0.91 → 0.14) while dimension 6 barely moves (0 → 0.001 → 0.002 → 0.003) — exactly the fast-hand / slow-hand pattern promised above.

Visualising the full pattern

The heatmap below plots PE(pos, dim) for all 16 positions (rows) against all 8 dimensions (columns) of this same d_model = 8 setup, colour-coded from blue (−1) through white (0) to amber (+1). Read it column by column: the left two columns flip colour almost every single row (the fast hands), while the right two columns stay almost pure amber all the way down (the slow hands, barely past their starting angle after 15 steps).

Why sine and cosine — the trig identity that makes relative position learnable

Bounded values and multiple frequencies explain why sinusoids are a reasonable encoding. But the deeper reason the original paper chose this exact sin/cos pairing is that it makes relative position — "these two tokens are 3 apart" — extractable by a simple linear operation, which is exactly the kind of operation attention layers are built from (dot products and weighted sums).

Take a single frequency w and compute the dot product between the encoding at position pos and the encoding at position pos + k, using just that one (sin, cos) pair:

PE(pos)·PE(pos+k) = sin(w·pos)·sin(w·(pos+k)) + cos(w·pos)·cos(w·(pos+k))

Recall the cosine difference identity from trigonometry (the same identity used in JEE-level trig problems): cos(A − B) = cos A cos B + sin A sin B. Set A = w(pos+k) and B = w·pos. The right-hand side of the identity is exactly the expression above (multiplication commutes, so the order of the sin and cos terms doesn't matter), and A − B = wk. So:

PE(pos)·PE(pos+k) = cos(w · k)

The pos has completely cancelled out. The similarity between a position and another position k steps ahead depends only on the gap k, never on where in the sequence you start counting from. Token 4 and token 7 produce the same positional dot product as token 104 and token 107, because both pairs are 3 apart. This is precisely the property a good position signal needs: it lets the network learn rules like "attend strongly to the token 2 positions back," and that rule works everywhere in the sequence, not just at one specific spot.

The same fact can be stated as a linear transformation on the whole (sin, cos) pair, not just their dot product. Using the sine and cosine sum identities sin(A+B) = sin A cos B + cos A sin B and cos(A+B) = cos A cos B − sin A sin B with A = w·pos, B = w·k:

PE(pos+k, 2i)   = PE(pos,2i)·cos(wk) + PE(pos,2i+1)·sin(wk)
PE(pos+k, 2i+1) = PE(pos,2i+1)·cos(wk) − PE(pos,2i)·sin(wk)

In matrix form, this says [PE(pos+k,2i), PE(pos+k,2i+1)]^T = M_k · [PE(pos,2i), PE(pos,2i+1)]^T, where M_k is the 2×2 matrix [[cos(wk), sin(wk)], [-sin(wk), cos(wk)]] — an orthogonal matrix (determinant 1) representing a pure rotation by angle wk. Crucially, M_k depends only on the offset k, never on pos. This means: for any fixed offset, there is a single fixed rotation that carries the positional encoding at any position to the encoding k steps later, uniformly across the entire sequence. That is exactly the algebraic handle a linear attention layer can exploit to learn relative-position behaviour from a fixed additive signal — which is a remarkable amount of structure to get from a formula containing no learned parameters at all.

Why base 10000, specifically?

The base sets how wide a range of wavelengths the encoding spans. With base 10000 and typical model sizes (the original paper used d_model = 512, giving 256 frequency pairs), wavelengths range from about 2π ≈ 6 positions up to about 2π × 10000 ≈ 63,000 positions. That range comfortably covers everything from "the previous word" to "a position tens of thousands of tokens away," which is roughly the working range of sequence lengths transformers are trained on. A smaller base would compress the useful range and cause faraway positions to look aliased (their encodings would start repeating, since sine and cosine are periodic); a much larger base would waste most dimensions on wavelengths far longer than any sequence the model will ever see. 10000 is a design choice, tuned empirically for realistic sequence lengths — not a value derived from some deeper mathematical necessity.

Verifying the formula in code

Here is a direct NumPy implementation. Trace it against the hand-computed table above to confirm they agree exactly.

import numpy as np

def positional_encoding(seq_len, d_model, base=10000):
    PE = np.zeros((seq_len, d_model))
    positions = np.arange(seq_len).reshape(-1, 1)       # shape (seq_len, 1)
    i = np.arange(d_model // 2).reshape(1, -1)           # shape (1, d_model/2)
    angle_rates = 1.0 / np.power(base, (2 * i) / d_model)
    angles = positions * angle_rates                     # shape (seq_len, d_model/2)
    PE[:, 0::2] = np.sin(angles)
    PE[:, 1::2] = np.cos(angles)
    return PE

pe = positional_encoding(seq_len=4, d_model=8)
print(np.round(pe, 4))

Trace it: positions = [[0],[1],[2],[3]], i = [[0,1,2,3]], so angle_rates = 1 / 10000^(2i/8) = [1, 0.1, 0.01, 0.001]. Broadcasting positions * angle_rates gives row pos=2 as [2, 0.2, 0.02, 0.002]; taking sin of that row and placing it in the even columns, cosine in the odd columns, reproduces the pos=2 row from the hand table exactly: [0.9093, -0.4161, 0.1987, 0.9801, 0.0200, 0.9998, 0.0020, 1.0000]. The printed output for all four rows matches the worked table above, digit for digit.

Correcting a second misconception: what the vector actually "means"

It is tempting to think of PE(pos) as something like a GPS coordinate the network "reads off" to learn its absolute position — as if the model computes "oh, this is token number 47." That is not how it gets used. The encoding is added directly to the word embedding, so the network never sees a clean, separable "position number" — it sees one blended vector. What actually carries the useful signal is relationships between positional vectors: the dot-product-depends-only-on-k property proved above, and the fixed-rotation property that follows from it. Attention layers work by comparing vectors (via dot products) and combining them (via weighted sums) — operations that are exactly suited to extracting relative-offset information out of these relationships, not to reading off an absolute index like a coordinate on a map. The model learns to use position differences, not position labels.

Sinusoidal vs. learned positional embeddings — and where the field went next

The sinusoidal formula above is one of two classic approaches. The alternative, used in BERT and GPT-2, is a learned positional embedding: simply a trainable lookup table with one vector per position, initialised randomly and updated by gradient descent just like word embeddings. Learned embeddings can, in principle, fit whatever positional pattern the training data actually needs, since nothing about their shape is fixed in advance. But they have a hard ceiling: if the table has 512 rows because training sequences never exceeded 512 tokens, there is no row 513 — the model simply cannot process a longer input without retraining or ad hoc extrapolation tricks. The sinusoidal formula has no such ceiling: plug in pos = 10000 and it produces a perfectly well-defined vector, because it's a formula, not a lookup table.

Neither approach is what most large language models use today. A more recent method called Rotary Position Embedding (RoPE, introduced by Su and colleagues in 2021, and used in models such as LLaMA and Mistral) takes the rotation-matrix idea derived above and applies it directly to the query and key vectors inside each attention layer — rotating them by an angle proportional to position — rather than adding a separate positional vector to the input embedding beforehand. It is, in a real sense, a direct engineering descendant of the exact M_k rotation matrix property this chapter just derived: once you notice that sinusoidal encodings behave like rotations under position shifts, building the rotation into the attention computation itself, rather than bolting it onto the embedding, is the natural next step.

Exam-relevant connections

• Trigonometric identities (angle sum and difference formulas) are the entire engine behind the relative-position derivation above — this is the same toolkit tested in JEE/BITSAT trigonometry, applied to a real research result instead of an abstract identity to memorise.
• Matrices and linear transformations: the rotation matrix M_k is orthogonal with determinant 1 — a standard object in JEE/BITSAT matrix chapters, here appearing as the mechanism inside a transformer.
• Permutation equivariance — proving Attention(PX) = P·Attention(X) is exactly the style of "prove this operator has property X" question that appears in GATE-foundation-level linear algebra and functions.
• The RNN-vs-transformer parallelism trade-off (sequential recurrence encodes order automatically; parallel attention does not) is a common conceptual question in board-level "emerging technologies in AI" sections and viva questions.

Active recall

1. Using the d_model=8 table's frequencies (denominators 1, 10, 100, 1000), compute PE(5, 0) and PE(5, 1) by hand. (Answer: sin(5) ≈ -0.9589, cos(5) ≈ 0.2837.)
2. True or false: if you reverse the order of every word in a sentence and feed it through a self-attention layer with no positional encoding added, each individual token's output vector changes. Justify using the permutation-equivariance argument from this chapter. (Answer: false — each token's output value is unchanged; only which row of the output holds it changes, exactly as shown in the Ravi/Meera example.)
3. A model was trained only on sequences up to 512 tokens using learned positional embeddings. Explain concretely why it cannot process a 600-token input, while a model using the sinusoidal formula can. (Answer: the learned table has no row for positions 512–599; the sinusoidal formula is a closed-form function of pos and is defined for any integer.)
4. Derive, from the angle-sum identities, why PE(pos)·PE(pos+k) for a single frequency pair equals cos(wk) and not some function that also involves pos. (Answer: it is a direct application of the cosine-difference identity cos(A-B)=cos A cos B + sin A sin B, shown step by step above — the pos terms cancel because A - B = wk regardless of the common offset added to both A and B.)

Summary

Self-attention, on its own, is permutation-equivariant: shuffle the input tokens and the outputs shuffle identically in value, never in content — a fact provable directly from the dot-product-softmax-weighted-sum structure of the attention formula, and verified numerically above with the Ravi/Meera example. Because transformers process all tokens in parallel rather than recurrently, this order-blindness has to be fixed by explicitly injecting position information before attention runs. Raw indices fail because they are unbounded and don't generalise to unseen lengths; length-normalised indices fail because a fixed relative gap means different things in sequences of different lengths. The sinusoidal solution assigns each position a vector of sine/cosine values across a geometric range of frequencies — like a bank of clock hands running from fast to extremely slow — keeping every value bounded in [-1, 1] and giving every position a unique fingerprint. The angle-sum and angle-difference identities prove this fingerprint has an exact, powerful property: the dot product and the linear transformation relating any two positions depend only on their distance apart, never on their absolute location, which is exactly the structure attention needs to learn relative-position behaviour from a fixed, parameter-free signal. That same rotation idea, applied more directly inside the attention computation itself, is the basis of Rotary Position Embedding, used in many of today's large language models.

Think About It

Think about this: How would you explain positional encoding: teaching order to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where positional encoding: teaching order is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting positional encoding: teaching order to at least 3 other topics you have studied.