RAG: Retrieval-Augmented Generation

Ask a large language model, "What is the last date to submit my Class 12 CBSE practical marks this year?" and watch it do something dangerous: it will answer confidently, fluently, and possibly wrongly. The model was trained months or years ago on text scraped from the internet. It never saw your school's circular. Yet it will not say "I don't know." It will invent a plausible-sounding date, because a language model's job is to produce the most likely next words, not the most true ones. This gap between fluency and truth is the single biggest obstacle to trusting AI, and Retrieval-Augmented Generation (RAG) is the most important engineering answer we have to it.

Here is the core idea in one sentence, and the rest of this chapter unpacks it: instead of asking the model to answer from memory, we first fetch the relevant documents, then ask the model to answer using those documents. We turn a closed-book exam into an open-book exam.

The closed-book vs open-book analogy

Think about two students sitting for a History paper on the Revolt of 1857. Student A memorised the whole textbook and must recall everything under pressure — she is a plain language model. When she forgets a date, she guesses, and her guess sounds just as confident as her correct answers. Student B is allowed to bring the textbook and, crucially, knows exactly which page to flip to before writing each answer. Student B is a RAG system. Neither student is "smarter" in raw language ability; the difference is that B grounds every answer in a passage she just re-read.

Notice what RAG is not. It is not retraining or fine-tuning the model on your documents — that is expensive, slow, and has to be redone every time a document changes. RAG leaves the model's weights completely untouched. It changes only what we put into the prompt at question time. This is why a RAG system can answer a question about a circular that was published five minutes ago, while a fine-tuned model would need hours of retraining. The model is a fixed reasoning engine; RAG feeds it fresh, relevant facts on demand.

Two halves: Retriever and Generator

Every RAG system is literally two systems bolted together, and the name tells you the order: Retrieval, then Augmented Generation.

• The Retriever takes the user's question and searches a collection of documents for the few passages most likely to contain the answer. Its output is text.
• The Generator (the language model) takes the question plus those retrieved passages and writes the final answer, instructed to rely on the passages rather than its own memory.

The whole pipeline, end to end, looks like this:

How does a computer decide two texts are "similar"? Embeddings

The retriever's whole job rests on one question: given a query, which document passages are about the same thing? Keyword matching is too brittle — a passage about "the deadline for submitting internal assessment" is highly relevant to "last date to submit practical marks," yet they share almost no words. We need to compare meaning, not letters.

The trick is embeddings. An embedding model maps any piece of text to a point in a high-dimensional space — a vector of numbers, typically 384, 768, or 1536 of them. It is trained so that texts with similar meaning land close together and unrelated texts land far apart. "Deadline" and "last date" end up as nearby points; "deadline" and "photosynthesis" end up far apart. Real embeddings have hundreds of dimensions, but the geometry is identical to what you can picture in 3D, so we will work an example in 3 dimensions where you can actually see the numbers.

Measuring closeness: cosine similarity, derived

Once texts are vectors, "similar meaning" becomes "small angle between vectors." We do not usually use straight-line (Euclidean) distance, because a long document and a short one can point in exactly the same direction but have very different lengths. What we care about is direction, and the natural measure of directional agreement is the cosine of the angle between two vectors.

Recall the dot product from your Class 11 vectors chapter. For two vectors a and b, the dot product is defined geometrically as:

a · b = |a| |b| cos(θ)

where θ is the angle between them and |a| is the magnitude (length) of a. Rearranging for the cosine directly gives us our similarity score:

              a · b            Σ aᵢ bᵢ
cos(θ) = ───────────── = ─────────────────────
            |a| |b|        √(Σ aᵢ²) · √(Σ bᵢ²)

This is cosine similarity. Trace what it does. The numerator, the dot product Σ aᵢbᵢ, is large and positive when the two vectors have big values in the same coordinates — when they "agree" about which dimensions matter. The denominator divides out the lengths of both vectors, so a passage does not score higher just for being long. The result always lies in [−1, 1]: it is 1 when the vectors point the same way (θ = 0, identical meaning), 0 when they are perpendicular (θ = 90°, unrelated), and negative when they point in opposing directions. For text embeddings the values are almost always between 0 and 1, and higher means more relevant.

Worked example: retrieving the right chunk

Suppose a student in our tutor app asks: "How do plants make food?" The embedding model turns that query into the vector q = [0.9, 0.1, 0.4]. (Imagine, loosely, the three axes mean "biology / process," "sports," and "food / energy.") Our little knowledge base has three chunks, already embedded:

• d₁ — photosynthesis note: [0.6, 0.1, 0.4]
• d₂ — generic syllabus page ("Unit 4 covers plant biology, sports day, and the mid-day meal scheme"): [0.6, 0.3, 0.4]
• d₃ — cricket match report: [0.1, 0.7, 0.1]

Let us compute cos(θ) for the photosynthesis chunk by hand, showing every step. First the dot product:

q · d₁ = (0.9)(0.6) + (0.1)(0.1) + (0.4)(0.4)
       = 0.54 + 0.01 + 0.16
       = 0.71

Now the magnitudes:

|q|  = √(0.9² + 0.1² + 0.4²) = √(0.81 + 0.01 + 0.16) = √0.98  ≈ 0.9899
|d₁| = √(0.6² + 0.1² + 0.4²) = √(0.36 + 0.01 + 0.16) = √0.53  ≈ 0.7280

cos(θ) = 0.71 / (0.9899 × 0.7280) = 0.71 / 0.7207 ≈ 0.985

The photosynthesis chunk scores 0.985 — a very small angle, almost perfect alignment. Now do the syllabus chunk d₂, which is the interesting one:

q · d₂ = (0.9)(0.6) + (0.1)(0.3) + (0.4)(0.4) = 0.54 + 0.03 + 0.16 = 0.73
|d₂|   = √(0.6² + 0.3² + 0.4²) = √(0.36 + 0.09 + 0.16) = √0.61 ≈ 0.7810
cos(θ) = 0.73 / (0.9899 × 0.7810) = 0.73 / 0.7731 ≈ 0.944

And the cricket report d₃ scores about 0.283 (you can verify this the same way). So the ranking is 0.985, 0.944, 0.283. The retriever would pick the photosynthesis note first — exactly what we want.

Here is the Python that does all three at once. Read it carefully; the printed output below is the actual output of running this code:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(x * x for x in b))
    return dot / (norm_a * norm_b)

query = [0.9, 0.1, 0.4]
chunks = {
    "photosynthesis": [0.6, 0.1, 0.4],
    "syllabus":       [0.6, 0.3, 0.4],
    "cricket":        [0.1, 0.7, 0.1],
}

scores = {name: round(cosine(query, vec), 3) for name, vec in chunks.items()}
ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

print("scores:", [scores[n] for n in chunks])
print("best chunk:", ranked[0][0])

scores: [0.985, 0.944, 0.283]
best chunk: photosynthesis

The syllabus chunk scoring 0.944 is a useful warning, not just a detail. It is almost as high as the true answer, even though it only mentions plant biology in passing among cricket and mid-day meals. A near-miss chunk that is topically adjacent but does not actually contain the answer is one of the most common ways a retriever goes wrong: it crowds the top-k list and can push a genuinely better chunk out. The smaller the true gap between "the right passage" and "a vaguely related passage," the more your whole RAG system depends on good chunking and good embeddings.

Chunking: why we do not embed whole documents

You cannot embed an entire 300-page NCERT textbook as one vector — the meaning would be an unusable average of everything, and you could never fit the whole book into the model's prompt anyway. So before anything else, RAG systems chunk: they split documents into passages of a few hundred words, embed each chunk separately, and store the chunk-vector pairs in a vector store (a database built to answer "give me the k nearest vectors to this query vector" quickly).

Chunking is a real engineering trade-off, and getting it wrong quietly wrecks retrieval:

• Chunks too large: each vector blends several topics, so its direction is muddy and cosine similarity becomes unreliable — exactly the syllabus-chunk problem, amplified.
• Chunks too small: a single sentence may lose the context that makes it meaningful. "It is due on the 15th" is useless if the sentence naming what is due sits in a different chunk.
• A common fix is overlapping chunks — for example, 300-word chunks that share their last 50 words with the next chunk — so a fact straddling a boundary survives in at least one complete chunk.

The "Augmented" step: building the prompt

Once the top-k chunks are retrieved, RAG assembles a new prompt. It does not just forward the chunks; it wraps them in an instruction that constrains the model. A typical template:

You are a tutor. Answer the question using ONLY the context below.
If the answer is not in the context, say "I don't have that information."

Context:
[chunk 1 text]
[chunk 2 text]

Question: How do plants make food?
Answer:

That instruction — "using ONLY the context" and "say I don't know" — is what converts an over-confident guesser into an honest, grounded answerer. The model now has the facts sitting right in front of it and explicit permission to admit ignorance. This is the whole reason RAG reduces hallucination.

A common misconception, corrected

Many students first meet RAG and conclude: "So the model reads all my documents and learns them." This is wrong, and the mistake matters. The language model never sees your document collection. It only ever sees the handful of chunks the retriever selected for this one question, pasted into this one prompt. Nothing is memorised, nothing is stored inside the model, and the next question starts from scratch. A direct consequence: if the retriever fails to surface the right chunk, the model cannot answer correctly no matter how intelligent it is — it was simply never shown the fact. In a RAG system, most "the AI is wrong" bugs are actually retrieval bugs, not generation bugs. Fixing them means fixing chunking, embeddings, or the number of chunks retrieved, not the model.

Failure modes worth naming

• Missed retrieval. The right chunk exists but ranks 11th when you only fetch the top 5. Cure: retrieve more candidates, then re-rank.
• Distractor chunks. This is why "the syllabus chunk scored 0.944" in our example is a genuine concern: a topically-close-but-wrong passage can outrank or crowd out the true answer, and the model may quote it.
• Context overflow. Prompts have a maximum length. Stuff in too many chunks and the earliest ones get truncated or ignored.
• Stale store. If a document changes but you never re-embed it, the retriever serves the old version — RAG is only as fresh as its last indexing run.

Where this shows up in India

RAG is the standard architecture behind almost every "chat with your documents" product you will meet. An IRCTC-style help bot that answers "what is the tatkal booking window for AC coaches?" should not guess from a model's stale memory; it should retrieve the current rule from the live policy document and answer from that — and update instantly when the rule changes, with no retraining. A bank building a UPI support assistant grounds answers in its own current FAQ so it never invents a transaction limit. A CBSE study app like this one can point a RAG retriever at the exact NCERT chapter a student is on, so answers are grounded in their syllabus rather than the whole internet. In every case the win is identical: current, checkable, source-grounded answers instead of confident fiction.

Exam mapping

For CBSE Informatics Practices and Computer Science, and for AI-elective and competitive-exam contexts, be ready to: (1) draw the retrieve-then-generate pipeline and label both halves; (2) explain why RAG reduces hallucination without retraining; (3) compute a cosine similarity by hand from the dot-product definition — a favourite because it ties directly to the Class 11–12 vectors syllabus; and (4) distinguish RAG from fine-tuning (RAG changes the prompt at query time; fine-tuning changes the weights ahead of time). The cosine derivation from a·b = |a||b|cos θ is the kind of "connect two chapters" step examiners love.

Active recall — do these, don't just read them

1. By hand, compute the cosine similarity between q = [0.9, 0.1, 0.4] and the cricket chunk [0.1, 0.7, 0.1]. Show the dot product and both magnitudes, and confirm you get ≈ 0.283.
2. A new chunk embeds to [0.9, 0.1, 0.4] — identical to the query. What is its cosine similarity, and what angle does that correspond to? Explain in one line why identical direction gives that value.
3. Your RAG tutor answers a question about a rule that changed yesterday. It gives the old answer. Is this most likely a retrieval bug or a generation bug? Name the specific failure mode and one fix.
4. Explain to a friend, without using the word "vector," why RAG can answer a question about a document published five minutes ago but a fine-tuned model cannot.
5. You increase your chunk size from 300 words to 2000 words and retrieval quality drops. Using the syllabus-chunk example, explain precisely why.

Summary — the key ideas

• A plain language model answers from frozen memory and hallucinates confidently when it does not know. RAG turns the closed-book exam into an open-book one.
• RAG = Retriever (find the relevant passages) + Generator (answer using them). It changes the prompt, never the model's weights — so it needs no retraining and stays current.
• Text is compared by meaning via embeddings — vectors positioned so similar meanings sit close together.
• Closeness is measured by cosine similarity, cos θ = (a·b)/(|a||b|), derived straight from the dot product; it scores direction agreement in [−1, 1], ignoring length.
• Documents are chunked and stored in a vector store; chunk size is a real trade-off, and near-miss "distractor" chunks (our 0.944 syllabus chunk) are a genuine hazard.
• Most RAG errors are retrieval errors: if the right chunk is never fetched, no amount of model intelligence can save the answer.