Sentence Embeddings: Whole Text as Vector

Picture a scholarship-helpdesk chatbot on a school's app. One student types "my scholarship money hasn't come yet." Another types "I haven't received my stipend." A third types "when will the amount be credited?" All three are asking exactly the same thing, and a human reading them instantly knows it. But these three sentences share almost no words in common — "money" versus "stipend" versus "amount," "come" versus "received" versus "credited." If the chatbot is matching queries to FAQ answers by comparing words directly, it fails all three. What it actually needs is a way to compare meaning, not spelling. That means turning each entire sentence into a single point in space — a vector — such that sentences meaning the same thing land near each other, no matter how differently they're worded. That single vector, and how to build it correctly, is what this chapter is about.

Why a Single Word Vector Isn't Enough

You've already seen that individual words can be represented as vectors — points in a space of, say, 100 or 300 dimensions — trained so that words used in similar contexts end up with similar vectors (this is the distributional hypothesis: "you shall know a word by the company it keeps"). "Delhi" and "Mumbai" end up close together; "happy" and "joyful" end up close together; "happy" and "algebra" end up far apart.

The trouble starts the moment you have a full sentence instead of a single word. A word embedding gives you one vector per word — so the sentence "I love cricket" gives you three separate vectors, one each for "I," "love," and "cricket." A five-word sentence gives you five vectors. A twenty-word sentence gives you twenty. But almost everything you'd want to do with sentences — search for the closest matching FAQ, cluster similar reviews, detect duplicate questions, feed a sentence into a classifier — needs a fixed number of inputs. You cannot compare a 3-vector sentence to a 20-vector sentence directly, and you cannot feed a variable-length list of vectors into an algorithm (like cosine similarity or logistic regression) that expects one vector of a fixed size. A sentence embedding is exactly this: a function that takes a variable-length sequence of word vectors and produces one fixed-length vector representing the whole sentence's meaning.

The Simplest Idea: Mean Pooling

The most direct fix is embarrassingly simple: average the word vectors. If a sentence has word vectors w₁, w₂, …, wₙ, define the sentence vector as

v_sentence = (w_1 + w_2 + ... + w_n) / n

This is called mean pooling, or a bag-of-embeddings representation, because — like the older bag-of-words model — it treats the sentence as an unordered bag of its words, except now the "bag" holds meaningful vectors instead of just word counts.

Let's test whether this actually captures meaning, with a tiny toy vocabulary of 3-dimensional vectors (real embeddings use 100–1000 dimensions; 3 is just small enough to compute by hand):

I       = [1, 0, 0]
love    = [0, 3, 1]
enjoy   = [0, 2.8, 1.2]
hate    = [0, -3, 1]
cricket = [2, 0, 2]
maths   = [2, 0, -2]

Notice "love" and "enjoy" were deliberately given nearly identical vectors (they're near-synonyms), while "hate" points in the opposite direction along that same axis — this is what a well-trained word-embedding model would produce on its own; here we're just setting it up by hand to see what happens next.

Averaging gives:

v("I love cricket")  = ([1,0,0]+[0,3,1]+[2,0,2]) / 3 = [1.000, 1.000, 1.000]
v("I enjoy cricket") = ([1,0,0]+[0,2.8,1.2]+[2,0,2]) / 3 = [1.000, 0.933, 1.067]
v("I hate maths")    = ([1,0,0]+[0,-3,1]+[2,0,-2]) / 3 = [1.000, -1.000, -0.333]

Now we need a way to measure how "close" two of these vectors are — not their raw distance, but the angle between them, since two sentences about the same topic should point in roughly the same direction in meaning-space regardless of exact magnitude. That measure is cosine similarity, and because the entire rest of this chapter leans on it, it's worth deriving properly rather than just stating it.

Deriving Cosine Similarity

For two vectors a and b in ℝⁿ, think of them as two arrows from the origin, with angle θ between them. The vector connecting their tips is a − b. Its squared length can be computed two ways.

First, algebraically, using the dot product identity |x|² = x·x:

|a - b|^2 = (a - b)·(a - b) = a·a - 2(a·b) + b·b = |a|^2 - 2(a·b) + |b|^2

Second, geometrically, by the law of cosines applied to the triangle formed by a, b, and a − b:

|a - b|^2 = |a|^2 + |b|^2 - 2|a||b| cos(theta)

Both expressions equal |a − b|², so set them equal and cancel |a|² + |b|² from both sides:

-2(a·b) = -2|a||b| cos(theta)
a·b = |a||b| cos(theta)
cos(theta) = (a·b) / (|a| |b|)

where a·b = Σᵢ aᵢbᵢ (dot product) and |a| = √(Σᵢ aᵢ²) (Euclidean norm, from the Pythagorean theorem extended to n dimensions). This ratio, cosine similarity, ranges from −1 (pointing exactly opposite) through 0 (perpendicular, i.e. unrelated) to +1 (pointing exactly the same way) — and crucially, it ignores vector length entirely, comparing only direction. That's exactly the property we want: a long, detailed sentence and a short paraphrase of it shouldn't be judged dissimilar just because one averaged more words than the other.

Applying this to our three sentence vectors (computing precisely, not by hand-rounding):

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

s1 = np.array([1.000, 1.000, 1.000])       # "I love cricket"
s2 = np.array([1.000, 0.933, 1.067])       # "I enjoy cricket"
s3 = np.array([1.000, -1.000, -0.333])     # "I hate maths"

print(round(cosine_similarity(s1, s2), 3))  # 0.999
print(round(cosine_similarity(s1, s3), 3))  # -0.132

"I love cricket" and "I enjoy cricket" — different words, same meaning — come out at cosine similarity 0.999, essentially identical in direction. "I love cricket" and "I hate maths" — different topic, opposite sentiment — come out at −0.132, close to unrelated. Mean pooling, despite its simplicity, has captured something real about meaning. This is exactly the mechanism a semantic search or duplicate-question system relies on: embed every FAQ once, embed the user's query, and rank FAQs by cosine similarity to the query vector.

The Diagram: What This Pipeline Looks Like

Where Mean Pooling Breaks: A Common Misconception

It's tempting to conclude, from the result above, that averaging word vectors basically "solves" sentence meaning. It doesn't — and knowing exactly why matters more than the technique itself.

Misconception: "If two sentences produce nearly the same vector, they mean nearly the same thing." This is false in general, and mean pooling is where it breaks most visibly, for two separate reasons.

Reason 1 — word order is thrown away entirely. Averaging is commutative: a + b + c gives the same sum regardless of the order you add them in. So "Ravi defeated Arjun in the final" and "Arjun defeated Ravi in the final" use the exact same multiset of words — only the positions of "Ravi" and "Arjun" are swapped — and mean pooling produces the identical vector for both, giving cosine similarity of exactly 1.000. Yet these two sentences report opposite winners. Mean pooling has no notion of subject versus object, cause versus effect, or "before" versus "after" — it only knows which words appeared, not how they relate to each other.

Reason 2 — small-magnitude function words get silently ignored. Extend the toy vocabulary with a few more words:

this  = [0.1, 0, 0]
movie = [2, 0, 0.5]
don't = [0, 0, 0.2]

"don't" is a real word with a real vector, but function words tend to occur in almost every kind of sentence regardless of topic, so their trained vectors typically end up short and directionless compared to strongly topical words like "love" or "cricket" — there's little consistent context for the training process to anchor them to. Averaging "I love this movie" gives:

v("I love this movie")        = [0.775, 0.750, 0.375]
v("I don't love this movie")  = [0.620, 0.600, 0.340]

cosine_similarity = 0.999

Adding the single word that completely reverses the sentence's meaning barely nudges the vector, because "don't" is just one small vector added to a sum of four (now five) terms, and its small magnitude means it can't pull the average far. The model rates this negated pair just as similar (0.999) as it rated the genuine synonym pair "love"/"enjoy" earlier. From mean pooling's point of view, negation might as well not exist. This is a real, well-documented limitation of bag-of-embeddings representations — not a toy-vocabulary artifact — and it's precisely why production systems reach for the fixes below rather than stopping at plain averaging.

A Partial Fix: TF-IDF-Weighted Averaging

Reason 2 above has a partial patch: instead of a plain average, weight each word's vector by how informative that word typically is, rather than treating every word equally. The classic weight is TF-IDF (term frequency–inverse document frequency), borrowed from information retrieval:

idf(w) = ln( N / df(w) )

where N is the number of documents (or sentences) in a reference corpus and df(w) is the number of those documents containing word w at least once. A word that appears in nearly every sentence gets an IDF close to zero; a word that appears in only a few gets a large IDF.

Take a 5-sentence toy corpus: "cricket is fun," "maths is hard," "the exam is tomorrow," "chess is fun," "coding is creative." Here N = 5, and "is" appears in all five sentences, so df("is") = 5:

idf("is")      = ln(5/5) = ln(1)   = 0.000
idf("fun")     = ln(5/2)           = 0.916   (appears in 2 of 5)
idf("cricket") = ln(5/1)           = 1.609   (appears in 1 of 5)

"is" gets an IDF of exactly zero — it contributes nothing to a weighted average, however important it looks grammatically. The weighted sentence vector is now

v_weighted = ( idf(w_1)*w_1 + idf(w_2)*w_2 + ... ) / ( idf(w_1) + idf(w_2) + ... )

For "cricket is fun," using the vectors cricket = [2, 0, 2], is = [0, 0, 0], fun = [0, 2.5, 1.5]:

plain average    = [0.667, 0.833, 1.167]
tfidf-weighted    = [1.274, 0.907, 1.819]

The weighted vector is pulled much more strongly toward "cricket" and "fun," the two words actually carrying the sentence's topic, and "is" — despite sitting grammatically at the centre of the sentence — is correctly given zero say in the result. This genuinely helps when function words dilute meaning. But notice what it does not fix: TF-IDF weighting still averages, so it's still completely blind to word order. Ravi and Arjun still swap identically. The negation problem is only partly helped too — "don't" wouldn't necessarily get a low IDF (it may well be a distinctive, informative word statistically), but weighting alone still can't express the relationship "don't negates love," because that relationship exists between two specific words, not in either word's vector alone. Fixing that requires abandoning simple averaging altogether.

Giving the Model a Sense of Order: Sequential Encoders

To make the sentence vector depend on word order, process the words one at a time, carrying forward a running summary. A recurrent neural network (RNN) does exactly this: starting from an initial hidden state h₀ (usually the zero vector), it updates a hidden state as each word vector xₜ arrives:

h_t = tanh( W_x . x_t + W_h . h_(t-1) + b )

Here Wₓ and Wₕ are weight matrices learned during training, and b is a learned bias. Crucially, hₜ depends on hₜ₋₁, which depended on hₜ₋₂, and so on back to h₀ — so the final hidden state h_T, taken as the sentence vector, has been shaped by the words in the exact order they arrived. Feed it "Ravi defeated Arjun" and it produces a genuinely different vector than "Arjun defeated Ravi," because by the time "defeated" is processed, the hidden state already "remembers" that "Ravi" came first. Plain RNNs struggle to carry information across long sentences (a problem called the vanishing gradient, where the influence of early words shrinks exponentially by the time the network reaches the end) — the LSTM and GRU architectures were designed specifically to fix this by adding gates that let the network choose what to keep and what to forget, but the core idea of a final hidden state acting as an order-sensitive sentence vector stays the same.

The Current State of the Art: Transformers and Sentence-BERT

Modern sentence embeddings mostly come from transformer models like BERT, which compute a contextual vector for every word using self-attention — each word's representation is built by weighing every other word in the sentence, not just processing left-to-right like an RNN. It's tempting to assume you can just mean-pool BERT's per-word output vectors, or take its special [CLS] summary token, and get an excellent sentence embedding for free. In practice this works poorly for comparing sentences by cosine similarity: raw BERT vectors are trained for a different objective (predicting masked words) and cluster together in ways that don't correlate well with human judgments of sentence similarity — a phenomenon researchers have documented and sometimes call anisotropy, where the vectors are squeezed into a narrow cone of the vector space rather than spread out to reflect meaning.

Sentence-BERT (SBERT), introduced by Nils Reimers and Iryna Gurevych in 2019, fixes this directly rather than hoping a pretrained model happens to be good at it. SBERT uses a siamese network structure: two copies of the same BERT model (sharing identical weights) each encode one sentence from a pair, mean-pool their outputs into fixed vectors, and the network is trained so that cosine similarity between the two resulting vectors matches whether the sentences actually mean similar things — using labelled sentence-pair datasets (natural language inference data, and human-annotated similarity scores) as the training signal. After this fine-tuning, cosine similarity between SBERT vectors reliably tracks semantic similarity, which plain BERT vectors do not. The practical payoff is speed: comparing every possible pair of sentences directly with full BERT (a "cross-encoder," which re-reads both sentences together for every comparison) needs one full model pass per pair — for 10,000 sentences that's roughly 50 million pairwise comparisons. The original SBERT paper reported this taking on the order of 65 hours of compute, versus about 5 seconds once each sentence has been embedded once into a fixed SBERT vector and pairs are compared with simple cosine similarity — because computing a dot product of two fixed vectors is essentially free compared to running a full transformer pass. This is the same reason semantic search engines can hold millions of pre-computed sentence vectors and search them near-instantly: the expensive encoding step happens once, offline, and querying is just fast vector arithmetic.

Putting the Pieces Together

Every technique in this chapter answers the same question — words to one fixed-length vector — with increasing sophistication: plain mean pooling (fast, order-blind, dilutable by function words), TF-IDF-weighted pooling (down-weights uninformative words, still order-blind), RNN/LSTM final hidden states (order-sensitive, but processes sequentially and can struggle over long sentences), and transformer-based encoders fine-tuned specifically for similarity like SBERT (contextual, order-sensitive, and directly optimized so that cosine similarity means what you want it to mean). This progression is a genuinely useful piece of code to have working end to end. Here is mean pooling plus cosine similarity, fully traceable:

import numpy as np

def sentence_vector(words, embeddings):
    vectors = [embeddings[w] for w in words]
    return np.mean(vectors, axis=0)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

embeddings = {
    "I": np.array([1, 0, 0]),
    "love": np.array([0, 3, 1]),
    "enjoy": np.array([0, 2.8, 1.2]),
    "hate": np.array([0, -3, 1]),
    "cricket": np.array([2, 0, 2]),
    "maths": np.array([2, 0, -2]),
}

s1 = sentence_vector(["I", "love", "cricket"], embeddings)
s2 = sentence_vector(["I", "enjoy", "cricket"], embeddings)
s3 = sentence_vector(["I", "hate", "maths"], embeddings)

print(round(cosine_similarity(s1, s2), 3))  # 0.999
print(round(cosine_similarity(s1, s3), 3))  # -0.132

Tracing it: sentence_vector looks up each word's vector and calls np.mean with axis=0, which stacks the list of 3-element arrays into a 3-by-n array and averages down each column — giving back a single 3-element vector, exactly the "add them up and divide by n" formula from earlier. cosine_similarity implements the derived formula directly: np.dot(a, b) computes Σaᵢbᵢ, and np.linalg.norm computes √(Σaᵢ²) for each vector. Running it reproduces the numbers computed by hand above.

Where This Shows Up

Sentence embeddings are the working machinery behind semantic search (matching a query to the closest-meaning document, not the closest-spelled one), duplicate-question detection (flagging that two differently-worded forum questions are asking the same thing), text clustering (grouping thousands of customer reviews or survey responses by topic without manual tagging), and as fixed-length inputs to downstream classifiers for tasks like sentiment analysis or topic labelling. If you're studying CBSE's Artificial Intelligence elective, you've likely already met bag-of-words and TF-IDF as ways of turning a document into numbers — sentence embeddings via weighted pooling are a direct extension of that same idea, just applied at sentence granularity and combined with trained word vectors instead of raw word counts. At the undergraduate and research level, this exact idea — encoding variable-length text into a fixed vector so that meaning can be compared by simple geometry — is foundational to the NLP portions of newer exams like GATE's Data Science and AI paper, and to essentially all serious work in information retrieval and applied NLP.

Check Your Understanding

1. Using the toy embeddings I = [1,0,0], hate = [0,-3,1], cricket = [2,0,2], compute the mean-pooled vector for "I hate cricket." (Answer: average of the three vectors = [1.000, -1.000, 1.000].)

2. Explain, without redoing any arithmetic, why "the dog chased the cat" and "the cat chased the dog" get the exact same mean-pooled vector. (Answer: both sentences contain the identical multiset of words — only their order differs — and averaging is unaffected by the order in which terms are summed, so the sums, and hence the means, are identical.)

3. A search engine mean-pools word vectors for "cheap flights to Chennai" and gets a poor match against "flights to Chennai that are not cheap." What specific weakness of mean pooling is responsible, and would switching to TF-IDF weighting fix it? (Answer: this is the negation/small-function-word problem — "not" is a short, low-magnitude vector whose contribution is swamped by content words like "cheap" and "flights," so the average barely shifts. TF-IDF weighting would not reliably fix this: "not" isn't necessarily rare across a corpus, and even if it were up-weighted, weighting still cannot express that "not" negates "cheap" specifically — that's a relationship between two words, which pooling of any kind discards. Only an order-sensitive encoder like an RNN or transformer can capture it.)

4. Why does an RNN's final hidden state make a better order-sensitive sentence vector than mean pooling, in one sentence? (Answer: because each hidden state hₜ is computed from the previous hidden state hₜ₋₁ and the current word, the final state has been shaped by the entire sequence in the order it arrived, not just the unordered set of words.)

5. Why can't you just take a pretrained BERT model's outputs, mean-pool them, and expect excellent sentence similarity scores — and what does SBERT change to fix this? (Answer: BERT is trained to predict masked words, not to make cosine similarity track sentence meaning, so its raw vectors cluster in ways — sometimes called anisotropy — that correlate poorly with human similarity judgments. SBERT fine-tunes a siamese pair of BERT encoders directly on labelled sentence-pair data so that cosine similarity between the pooled outputs is explicitly trained to match semantic similarity.)

Summary

A sentence embedding turns a variable-length sequence of word vectors into one fixed-length vector, so that whole sentences — not just words — can be compared by geometry. Mean pooling (averaging word vectors) is the simplest method and does capture real synonym relationships, measurable through cosine similarity, which is derived directly from the law of cosines as cos θ = (a·b)/(|a||b|). But plain averaging is provably blind to word order (swapping two words in an otherwise-identical sentence leaves the vector exactly unchanged) and tends to drown out short function words like negation — verified numerically above, where a sentence and its negation scored a 0.999 cosine similarity, just as high as two genuine synonyms. TF-IDF weighting fixes the "important words get diluted" problem by giving near-zero weight to words that appear in almost every sentence, but it does not fix order-blindness. Recurrent networks fix order-blindness by building the sentence vector one word at a time through a chain of hidden states. Transformer-based encoders fine-tuned specifically for similarity, like Sentence-BERT, currently give the strongest results, because they are trained end-to-end so that cosine similarity between two sentence vectors is a direct, calibrated measure of how similar the sentences actually mean — turning what would otherwise be an expensive pairwise comparison over the whole model into a single fast vector lookup.

Think About It

Think about this: How would you explain sentence embeddings: whole text as vector to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.