Generative AI and Large Language Models: The Future of AI

What Makes AI "Generative"?

Most of the AI you have already met in this course is discriminative: it looks at an input and picks one label from a fixed, closed set. A spam filter takes an email and outputs "spam" or "not spam." A digit-recognition model takes a scanned handwritten numeral and outputs one of ten digits. The output space is small and known in advance, and the model's job is to draw the correct boundary inside it.

A generative model does something structurally different: instead of choosing among a handful of pre-defined labels, it produces new content — a sentence, a paragraph, a line of code, an image — that did not exist before you asked for it. Formally, a generative model learns (an approximation of) the probability distribution over possible outputs, and then samples from that distribution to create something new. This is the entire family called Generative AI: image generators built on a technique called diffusion (which start from random noise and repeatedly denoise it, guided by your text prompt, until a coherent picture emerges), voice-cloning models that generate audio waveforms, and — the branch this chapter is about — Large Language Models (LLMs), which generate text one token at a time. Diffusion models and LLMs solve the "produce something new" problem with very different machinery (iterative denoising versus sequential next-token prediction), but they share the same generative logic: model a probability distribution over possible outputs, then sample from it. This chapter builds that logic rigorously for text, because the same idea, once you understand it precisely, transfers directly to why ChatGPT, GitHub Copilot, and India's own multilingual AI initiatives all work the way they do.

Your Keyboard Already Builds a Language Model

Open your phone's keyboard app and type "I am going to". It suggests words like "the," "school," or "be" above the keyboard — not "purple," not "quantum," not a random word. Somewhere inside that app is a small statistical model that has looked at enormous amounts of previously typed text and learned: after the words "going to," certain words are far more likely to come next than others.

That is, precisely, what a language model is. Formally: a language model is a function that takes a sequence of tokens $w_1, w_2, \ldots, w_{t-1}$ and outputs a probability distribution over what the next token $w_t$ could be — written $P(w_t \mid w_1, w_2, \ldots, w_{t-1})$. Your keyboard's autocomplete and the model behind ChatGPT are computing the exact same mathematical object. The gap between "occasionally useful for finishing a text message" and "writes working code and explains photosynthesis" is not a different idea — it is the same idea, trained on roughly a trillion times more text, restructured around one architectural upgrade called attention, which the rest of this chapter derives from first principles.

A Language Model You Can Compute By Hand: The Bigram Model

Before touching anything resembling a neural network, build the simplest possible language model and watch it work. A bigram model estimates $P(w_t \mid w_{t-1})$ — the probability of the next word given only the one word right before it — by counting how often each pair of consecutive words occurred in a training corpus.

Take this tiny four-sentence corpus:

india won the match
india won the series
pakistan won the toss
india lost the match

Count every consecutive word pair (bigram). The word "won" is followed by "the" in all three sentences where "won" appears, so $P(\text{the} \mid \text{won}) = 3/3 = 1.0$. The word "the" is followed by "match" twice, "series" once, and "toss" once, out of four total occurrences, so $P(\text{match}\mid\text{the})=0.5$, $P(\text{series}\mid\text{the})=0.25$, $P(\text{toss}\mid\text{the})=0.25$. This is not a metaphor for what the model does — it is literally counting and dividing. Here it is in code:

from collections import defaultdict, Counter

corpus = [
    "india won the match",
    "india won the series",
    "pakistan won the toss",
    "india lost the match",
]

bigram_counts = defaultdict(Counter)
for sentence in corpus:
    words = sentence.split()
    for i in range(len(words) - 1):
        bigram_counts[words[i]][words[i + 1]] += 1

def next_word_probs(word):
    counts = bigram_counts[word]
    total = sum(counts.values())
    return {w: c / total for w, c in counts.items()}

print(next_word_probs("the"))
print(next_word_probs("won"))

Tracing this by hand: bigram_counts["the"] is Counter({'match': 2, 'series': 1, 'toss': 1}), total 4, so the first print statement outputs {'match': 0.5, 'series': 0.25, 'toss': 0.25} (in that insertion order, since "match" is the first word ever seen after "the"). The second outputs {'the': 1.0}, matching the hand calculation exactly.

Now generate text by repeatedly picking the most likely next word (called greedy decoding) and feeding it back in:

prompt = "india"
generated = [prompt]

for step in range(3):
    probs = next_word_probs(generated[-1])
    next_word = max(probs, key=probs.get)
    generated.append(next_word)

print(" ".join(generated))

Trace it: start generated = ["india"]. Step 1: next_word_probs("india") gives {'won': 0.667, 'lost': 0.333} (won appears after india twice, lost once, out of three); the maximum is "won". Step 2: next_word_probs("won") gives {'the': 1.0}; append "the". Step 3: next_word_probs("the") gives match=0.5 as the maximum; append "match". Final output: india won the match — one of the exact training sentences, reproduced verbatim. This is an honest and important result, not a coincidence: a model this small, trained on this little data, has no way to say anything it hasn't essentially memorized. Real LLMs avoid pure repetition because they are trained on billions of documents spanning near-infinite combinations, so the "most likely next word" is rarely a verbatim copy of any one source — but the underlying generate-one-token-then-feed-it-back loop is identical to what you just traced by hand.

The bigram model's fatal weakness is exactly its definition: it only ever looks at one previous word. It has no way to use information from five or ten words earlier. That limitation is what the rest of this chapter exists to fix.

Tokens: How Text Actually Enters the Model

Real models don't operate on whole words the way the toy example did. They operate on tokens — pieces of text produced by an algorithm called subword tokenization (commonly Byte-Pair Encoding, BPE). Whole-word vocabularies break down fast: they can't represent a word they've never seen (a new brand name, a typo, a rare technical term), and every language would need its own enormous separate vocabulary. BPE instead starts from individual characters and repeatedly merges the most frequent adjacent pairs, building up a vocabulary of tens of thousands of frequently-recurring chunks — common short words stay whole ("the", "is"), while rare or compound words get split into pieces ("tokenization" might become "token" + "ization").

This matters concretely for Indian users, because a huge amount of real text is code-mixed — Hindi and English (or another Indian language and English) inside the same sentence, written in Roman script: "kal exam hai, please thoda help kar do." A subword tokenizer trained on enough code-mixed text learns to split this into a mix of recognizable English tokens and recognizable transliterated-Hindi subword pieces, rather than failing outright the way a rigid whole-word English dictionary would. This is precisely why building genuinely useful Indian-language AI is a tokenization and training-data problem as much as it is an "intelligence" problem — a model can only be as fluent in a script or code-mixing pattern as its tokenizer and training corpus allow.

Embeddings: Meaning as Geometry

Once text is split into tokens, each token is converted into a vector of real numbers, called an embedding. The core idea: tokens with related meaning get vectors that point in similar directions, and unrelated tokens get vectors that point in different directions. "Similar direction" is measured with cosine similarity: for two vectors $a$ and $b$, $$\text{cosine\_sim}(a,b) = \frac{a \cdot b}{\lVert a \rVert \lVert b \rVert}$$ where $a\cdot b$ is the dot product and $\lVert a \rVert$ is the vector's length. This ranges from $-1$ (opposite direction) to $1$ (identical direction), and unlike raw distance, it ignores vector magnitude and cares only about direction — which turns out to be where trained models store meaning.

You can see this concretely with the simplest possible embedding scheme: represent each short document as a vector counting which words from a fixed vocabulary it contains (a "bag of words"). Take three sentences:

import math

vocab = ["india", "wins", "the", "cricket", "match", "series", "stock", "market", "fell", "sharply"]

def vectorize(sentence):
    words = sentence.split()
    return [1 if v in words else 0 for v in vocab]

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(y * y for y in b))
    return dot / (mag_a * mag_b)

doc_a = vectorize("india wins the cricket match")
doc_b = vectorize("india wins the cricket series")
doc_c = vectorize("the stock market fell sharply")

print(round(cosine_similarity(doc_a, doc_b), 2))
print(round(cosine_similarity(doc_a, doc_c), 2))

Trace it: doc_a = [1,1,1,1,1,0,0,0,0,0], doc_b = [1,1,1,1,0,1,0,0,0,0], doc_c = [0,0,1,0,0,0,1,1,1,1]. For $A$ and $B$: dot product $=1+1+1+1=4$ (they share "india","wins","the","cricket"), $\lVert A\rVert=\lVert B\rVert=\sqrt5$, so cosine similarity $=4/5=0.8$. For $A$ and $C$: dot product $=1$ (only "the" is shared), $\lVert C\rVert=\sqrt5$, so cosine similarity $=1/5=0.2$. The program prints 0.8 then 0.2. Two sentences about cricket sit far closer together in this vector space (0.8) than a cricket sentence and a stock-market sentence (0.2) — even though "the" appears in all three, a single shared common word barely moves the similarity, while several shared content words move it a lot. This is the geometric intuition every embedding, however large, is built on.

Real embeddings, learned by training a neural network on billions of words rather than hand-built like the bag-of-words example, capture far richer structure. A famous empirical finding from early word-embedding research (word2vec, 2013) is that simple vector arithmetic on trained embeddings approximately recovers meaningful relationships — the vector for "king", minus the vector for "man", plus the vector for "woman", lands close to the vector for "queen". This does not hold exactly for every word pair, and it is a property that emerges from training on huge text corpora, not something explicitly programmed — but it demonstrates that the geometry these vectors settle into genuinely encodes relationships like gender and royalty as directions in space, not just word co-occurrence.

The Problem Long-Range Context Creates

Consider this classic ambiguous sentence: "The trophy doesn't fit in the suitcase because it is too big." What does "it" refer to — the trophy or the suitcase? Every fluent reader instantly says "the trophy," because "too big" combined with "doesn't fit" only makes sense if the trophy, not the suitcase, is the oversized object. Now flip one word: "The trophy doesn't fit in the suitcase because it is too small," and "it" flips to mean the suitcase.

A bigram model has no chance here: it only ever looks one word back, and the word immediately before "it" is "because" in both sentences — identical, uninformative. Even the bag-of-words embedding from the previous section fails, because it only counts which words appear, throwing away their order and their relationships to each other entirely. Resolving "it" requires reaching all the way back across eight intervening words and weighing them against each other. This is exactly the problem that motivated the architecture behind every modern LLM: self-attention.

Self-Attention: Letting Every Token Look at Every Other Token

Self-attention lets each token in a sentence directly compare itself against every other token and decide how much each one matters for updating its own meaning. Mechanically, every token's embedding is used to produce three vectors: a query (what am I looking for?), a key (what do I have to offer?), and a value (the actual content I'll contribute if selected). To update the representation of "it," the model takes its query vector and compares it — via a dot product — against the key vector of every other token in the sentence. A higher dot product means "more relevant." These raw relevance scores are then turned into a proper probability distribution (weights that are all positive and sum to exactly 1) using the softmax function: $$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$$ The exponential guarantees every weight is positive, and dividing by the total guarantees they sum to 1 — turning arbitrary relevance scores into something that behaves exactly like a probability distribution, which can then be used to take a weighted average of every token's value vector.

Work through it numerically. Suppose that after computing query-key dot products, "it" scores 2.0 against "trophy" and 0.0 against "suitcase" (the model has learned, from training on huge amounts of text, that "too big...doesn't fit" patterns typically point back at the object that didn't fit). Apply softmax by hand:

import math

def softmax(scores):
    exps = [math.exp(s) for s in scores]
    total = sum(exps)
    return [round(e / total, 4) for e in exps]

print(softmax([2, 0]))

Trace it: $e^2 \approx 7.389056$, $e^0 = 1$, sum $\approx 8.389056$. Weight for trophy $= 7.389056/8.389056 \approx 0.8808$; weight for suitcase $= 1/8.389056 \approx 0.1192$. The program prints [0.8808, 0.1192]. In plain terms: the model resolves "it" as roughly 88% "the trophy" and 12% "the suitcase" — and this attention weight is what actually determines how much of the trophy's meaning gets blended into the updated representation of "it" as the sentence is processed. The diagram below shows this exact computation, with the thickness and darkness of each arc encoding the attention weight.

This is called self-attention because every token does this against every other token in the same sequence — "the," "trophy," "big," and every other word are all simultaneously computing their own attention weights over the whole sentence. Real models run many independent attention computations in parallel per layer (called "heads," each free to learn a different kind of relationship — one head might specialize in pronoun resolution, another in matching verbs to their subjects), and stack many such layers, so the representation of each token is repeatedly refined using ever-richer context from the rest of the sequence. This stacked-attention architecture, introduced in the 2017 paper "Attention Is All You Need" by researchers at Google, is called the Transformer, and it is the architecture underlying every major LLM in use today.

The same softmax also controls how the model turns its final probability distribution into actual generated text, via a parameter called temperature ($T$), which rescales the scores before applying softmax: $\text{softmax}(z_i/T)$. Low temperature sharpens the distribution toward the single most likely token (closer to greedy decoding); high temperature flattens it, making less-likely tokens more competitive and outputs more varied.

def softmax_t(scores, temperature=1.0):
    scaled = [s / temperature for s in scores]
    exps = [math.exp(s) for s in scaled]
    total = sum(exps)
    return [round(e / total, 4) for e in exps]

print(softmax_t([2, 0], temperature=0.5))
print(softmax_t([2, 0], temperature=2.0))

At $T=0.5$, the scaled scores become $[4, 0]$: $e^4\approx54.598$, sum $\approx55.598$, giving weights $[0.982, 0.018]$ — sharper than the original $[0.88, 0.12]$. At $T=2.0$, scaled scores become $[1, 0]$: $e^1\approx2.718$, sum $\approx3.718$, giving weights $[0.7311, 0.2689]$ — flatter. This is exactly the "temperature" slider you may have seen in AI tools: it is not a metaphor, it is this rescale-then-softmax operation, applied to real vocabulary-sized score vectors instead of this two-word toy example.

Why Position Still Matters

Notice something uncomfortable about the attention mechanism as described: comparing every token's query against every other token's key treats the sentence as a set, not a sequence — nothing in the raw dot-product computation inherently encodes which token came first. But order is often the entire meaning. "Dog bites man" and "man bites dog" contain the identical bag of words but describe opposite events; in Hindi, "Ram ne Shyam ko maara" (Ram hit Shyam) and "Shyam ne Ram ko maara" (Shyam hit Ram) swap who is the attacker and who is the victim using exactly the same words in a different order. Because self-attention alone can't distinguish these, every Transformer adds a positional encoding — extra information mixed into each token's embedding that records its position in the sequence — before any attention is computed. Only with position folded in does the model have any way to tell "Ram hit Shyam" apart from "Shyam hit Ram."

Stacking It Into a Transformer, and Training It at Scale

A full LLM stacks dozens of these attention-plus-processing layers on top of each other, refining every token's representation layer by layer, and finishes by converting the last layer's representation of the current position into a probability distribution over the entire vocabulary — exactly the $P(w_t \mid w_1,\ldots,w_{t-1})$ from the very first section, just computed by a vastly more capable function than bigram counting.

Training such a model is conceptually the same next-token-prediction task as the bigram example, just at enormous scale: feed the model real text (books, websites, code, and more), have it predict each next token, measure how "surprised" it was by the actual next token (this surprise is formalized as cross-entropy loss — lower when the model assigned high probability to the true next token), and adjust the model's internal numbers (its parameters) via an optimization procedure called gradient descent to reduce that surprise, repeated across enormous quantities of text. This stage is called pretraining. For a sense of scale: GPT-3, described in a 2020 research paper, had 175 billion trainable parameters — compare that to the handful of counts our bigram model needed. Model sizes and training techniques have diversified substantially since, but the training objective — predict the next token correctly, adjust parameters, repeat — has not changed.

A model trained purely to continue text is not automatically a helpful assistant — left alone, it will happily continue a question with more questions, because that is a statistically common pattern in raw internet text. To turn a raw text-continuation engine into something like ChatGPT, developers add a further stage: fine-tuning on examples of good question-answer behavior, often refined further using human feedback on which responses people actually prefer (a process called reinforcement learning from human feedback, RLHF). Pretraining teaches the model the statistics of language; fine-tuning and alignment teach it to use that statistical knowledge the way a helpful assistant would.

The Misconception: "The AI Is Looking It Up"

A very common and important misunderstanding is treating an LLM like a search engine that consults a live, verified database of facts before answering. It does not, by default. Everything an LLM "knows" is encoded as statistical patterns baked into its parameters during pretraining — a frozen snapshot of what appeared, and how often, in its training text. When you ask it a question, it is not retrieving a stored fact; it is generating the token sequence that its trained parameters make statistically most probable given your prompt, exactly the same mechanism as the bigram model choosing "match" after "the," just vastly more sophisticated.

This is precisely why LLMs hallucinate — state incorrect information fluently and confidently. If a specific fact (an obscure date, a legal citation, a precise statistic) was rare, absent, or contradictory in the training data, the model still has to output some next token, and it will produce whatever sequence its learned probabilities favour, with no built-in mechanism to flag "I am uncertain" or "I am guessing" unless it was specifically trained to do so. It is also why an LLM by default cannot know about anything that happened after its training data was collected. When a chatbot appears to "search the web," it is typically using a separate, different technique called retrieval-augmented generation (RAG): a search step retrieves real documents and inserts their actual text into the model's context window, and the model then predicts its response conditioned on that genuine retrieved text — grounding generation in real sources rather than relying purely on memorized statistics. For CBSE board answers, project citations, or any factual claim you take from an AI tool, this has a direct practical consequence: verify specific facts, numbers, and citations against a textbook or primary source before writing them into an answer sheet, exactly as you would double-check a fact recalled from memory rather than looked up.

Where This Shows Up in India

Because tokenization and embeddings operate at the subword level rather than requiring one rigid model per language, the same Transformer architecture can, given enough training data, learn to handle multiple Indian languages and Hindi-English code-mixing within a single model — this is the technical foundation behind India's Bhashini initiative (the government's National Language Translation Mission), which aims to make AI-driven translation and language tools work across India's many scheduled languages rather than only in English. It is also why a single customer-support chatbot for a UPI app or a banking app can plausibly handle a query typed in Hinglish without needing an entirely separate model per language: the tokenizer breaks the mixed input into subword pieces from a shared vocabulary, and the attention mechanism builds context across them regardless of which language each token came from.

Exam Mapping

CBSE's Artificial Intelligence skill subject, offered from Class 9 onward, already introduces natural language processing and chatbots at a conceptual level — this chapter is the rigorous, mathematically grounded version of that same unit, giving you the actual probability, vector, and softmax computations behind the concepts you're asked to describe qualitatively there. Generative AI and LLMs are not tested directly in JEE's physics/chemistry/mathematics syllabus, but the probability distributions, vector dot products, and exponential functions used throughout this chapter are core Class 11–12 mathematics, directly useful background if you pursue Computer Science or Informatics Practices as a board subject, where programming problems involving frequency counting, vectors, and probability are standard. Increasingly, computational-thinking and AI-literacy questions in Olympiad-style and aptitude assessments test exactly the kind of reasoning this chapter builds: given a probability distribution or a similarity score, compute or interpret it correctly, and reason about a model's real capabilities and limitations rather than popular myths about it.

Active Recall Practice

• Extend the bigram corpus with the sentence "india won the toss" and recompute next_word_probs("the") by hand. (Answer: counts become match=2, series=1, toss=2, total=5, so probabilities are 0.4, 0.2, 0.4.)
• Using the bag-of-words vocabulary and vectorize function above, compute the cosine similarity between "india wins the cricket series" and "the stock market fell sharply." (Answer: only "the" is shared, dot product = 1, both magnitudes are $\sqrt5$, so cosine similarity = 0.2, identical to the doc_a–doc_c case, since doc_b and doc_c also share exactly one word.)
• Apply softmax by hand to scores [1, 1, 1] for three equally-relevant candidate tokens. (Answer: all three exponentials equal $e^1$, so each weight is exactly $1/3 \approx 0.3333$ — equal scores always produce a uniform distribution.)
• Explain, in one or two sentences, why a bag-of-words model would assign identical vectors to "the dog bit the man" and "the man bit the dog," and why this is a real limitation, not a minor detail.
• A classmate says, "ChatGPT is basically a search engine that finds the right answer in its database." Identify exactly what is wrong with this claim and correct it using the vocabulary from this chapter (parameters, pretraining, next-token prediction, hallucination, retrieval-augmented generation).

Summary

• A language model assigns a probability distribution $P(w_t \mid w_1,\ldots,w_{t-1})$ over the next token given everything before it; a bigram model estimates this by simple counting, and is computable entirely by hand.
• Generative AI is the broader family of models that sample new content from a learned probability distribution (diffusion models for images, LLMs for text); LLMs are the text-specific, token-by-token case.
• Text is broken into subword tokens (via Byte-Pair Encoding), not whole words, which is what lets one model handle rare words and code-mixed languages like Hinglish.
• Tokens become embedding vectors, and cosine similarity between vectors measures meaning-similarity; this was verified numerically with a bag-of-words example (0.8 for two cricket sentences vs. 0.2 for a cricket-vs-stock-market pair).
• Bigram and bag-of-words models both fail on long-range dependencies (like resolving "it" across eight words); self-attention fixes this by letting every token compute relevance scores (query-key dot products) against every other token, normalized into weights via softmax, exactly computed here as 0.88/0.12.
• Positional encoding is added because attention alone cannot tell word order apart, and order changes meaning ("dog bites man" vs. "man bites dog").
• Stacked attention layers form the Transformer architecture; LLMs are pretrained on massive text corpora via next-token prediction (GPT-3: 175 billion parameters) and then fine-tuned/aligned (e.g., via RLHF) to behave as helpful assistants.
• LLMs generate text autoregressively, one token at a time, feeding each generated token back as new context; temperature controls how sharply or flatly the next-token distribution is sampled.
• LLMs do not consult a live fact database by default — they generate statistically probable continuations from frozen training-time patterns, which is why they hallucinate; retrieval-augmented generation (grounding responses in real retrieved documents) is the actual mechanism behind "AI that searches the web." Always verify specific facts independently, especially for exam answers.

Think About It

Think about this: How would you explain generative ai and large language models: the future of ai to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.