Optimization Algorithms: How AI Learns Efficiently

The Problem: One Model, Millions of Unknowns

Every AI model — from a simple line that predicts exam scores to a large language model with billions of internal settings — is really just a mathematical function with a bunch of adjustable numbers plugged into it. Those adjustable numbers are called parameters (often called weights). "Training" a model means finding the specific values of those parameters that make the model's predictions match reality as closely as possible.

Here is the part that should bother you: nobody hands the model the right values. It has to find them itself, by trial and error, guided by math. For a model with two or three parameters, you could imagine searching by hand. For a model with a million parameters — a small neural network by modern standards — there is no "by hand." This chapter is about the algorithm that makes that search possible: gradient descent, and the family of optimization algorithms built on top of it. By the end, you will be able to derive the update rule yourself, trace it step by step on real numbers, and explain exactly why training sometimes explodes instead of converging.

A Familiar Starting Point: Fitting a Line by Hand

Suppose you want a tiny AI model that predicts a student's Class 10 mathematics test score (out of 100) from the number of hours they spent practising problems that week. You collect four data points:

• 1 hour → scored 12
• 2 hours → scored 19
• 3 hours → scored 31
• 4 hours → scored 40

Let's use the simplest possible model: predicted_score = w × hours, where w is a single number the model has to learn (we're deliberately leaving out an intercept term for now, to keep the algebra clean — we'll add it back later). If w = 10, the model predicts 10, 20, 30, 40 — close, but not exact.

To make "close" precise, we need a loss function: a number that measures how wrong the model's predictions are, so that small loss = good model. The standard choice for this kind of problem is Mean Squared Error (MSE):

L(w) = (1/n) × Σ (w·x_i − y_i)²

where x_i is hours studied, y_i is the actual score, and n is the number of data points. We square each error before averaging for two solid reasons, not just convention: squaring makes every error positive (so a +5 error and a −5 error don't cancel out and hide the true amount of wrongness), and it is differentiable everywhere — including at an error of exactly zero — which matters a great deal in a few paragraphs, because our whole strategy depends on being able to take a derivative. The absolute-error alternative, |w·x_i − y_i|, has a sharp corner at zero where no derivative exists, which makes it a worse fit for the algorithm we're about to build.

Why the Loss Function Is Shaped Like a Bowl

Let's expand L(w) algebraically for our four data points. First compute three sums from the data: Σx_i² = 1+4+9+16 = 30, Σx_i·y_i = 12+38+93+160 = 303, and Σy_i² = 144+361+961+1600 = 3066. Expanding the square inside the sum:

L(w) = (1/n)[w²·Σx_i² − 2w·Σx_iy_i + Σy_i²] = (1/4)[30w² − 606w + 3066] = 7.5w² − 151.5w + 766.5

That is a quadratic in w — the same kind of expression you've solved for years by completing the square. Doing exactly that:

L(w) = 7.5(w − 10.1)² + 1.425

This tells us everything about the shape instantly: it's an upward-opening parabola (bowl), its lowest point sits at w* = 10.1, and the minimum possible loss is 1.425. You already know how to find the minimum of a quadratic algebraically — you've been doing it since you learned to complete the square. So why do we need an entire algorithm?

Because this trick only works because our model has exactly one parameter and a squared-error loss, which happens to expand into a clean quadratic. A real neural network's loss function depends on thousands or millions of parameters simultaneously, and it is almost never a simple quadratic — it can twist, curve, and have complicated shapes that no algebra formula can solve directly. We need a method that finds the bottom of the bowl without needing a closed-form formula. That method is gradient descent, and testing it against a case we can already solve by algebra (like this one) is exactly how we'll verify it actually works.

A 60-Second Primer on the Derivative

If you haven't formally met calculus yet, here is the one idea you need. For a function f(w), the derivative f'(w) at a point tells you the slope of the curve at exactly that point — how fast f changes if you nudge w by a tiny amount. It's defined using a limit:

f'(w) = lim (h→0) [f(w+h) − f(w)] / h

Let's derive the one rule we'll actually need, for f(w) = w²:

[f(w+h) − f(w)] / h = [(w+h)² − w²] / h = [w² + 2wh + h² − w²] / h = [2wh + h²] / h = 2w + h

As h shrinks toward 0, 2w + h shrinks toward 2w. So d/dw(w²) = 2w. That's the entire "power rule" derived from scratch, not asserted. We'll also need one more fact, the chain rule: if you have a function-of-a-function, like e(w)² where e itself depends on w, its derivative is 2·e(w)·e'(w) — the outer power rule, multiplied by the derivative of what's inside. You can verify this is consistent with what we just derived by setting e(w) = w, giving e'(w)=1 and recovering 2w.

Deriving the Gradient of the Loss

Now apply this to our actual loss function. Define the error (residual) for data point i as e_i(w) = w·x_i − y_i. Then L(w) = (1/n)·Σ e_i(w)². Differentiating term by term using the chain rule:

dL/dw = (1/n)·Σ [2·e_i(w)·e_i'(w)]

Since e_i(w) = w·x_i − y_i, and x_i, y_i are fixed numbers from the data, e_i'(w) = x_i. Substituting:

dL/dw = (2/n) · Σ (w·x_i − y_i) · x_i

This is the gradient of the loss with respect to w — a formula, not a number, because it depends on the current value of w. It tells you the slope of the bowl at whatever point you're currently standing. If the slope is negative, the bowl goes downhill as w increases, so you should increase w. If positive, decrease w. That single sentence is the entire idea behind gradient descent.

Gradient Descent — the Algorithm, Traced Step by Step

The update rule is:

w_new = w_old − α · dL/dw

where α (alpha) is the learning rate, a small positive number you choose that controls how big each step is. Notice the minus sign — this is the detail students most often get backwards.

Common misconception, corrected: "Gradient descent moves in the direction the gradient points." This is false. The gradient (derivative) at a point points in the direction of steepest increase of the loss. To go downhill, you must move in the opposite direction — hence w_new = w_old − α·(gradient), not plus. If you ever see a plus sign in a gradient descent update without an explanation, something is wrong.

Let's trace this by hand, starting at w0 = 8 with learning rate α = 0.01. Here's the loop as code:

x = [1, 2, 3, 4]
y = [12, 19, 31, 40]
w = 8.0
lr = 0.01
n = len(x)

for step in range(4):
    grad_sum = 0
    loss_sum = 0
    for xi, yi in zip(x, y):
        error = w * xi - yi
        grad_sum += error * xi
        loss_sum += error ** 2
    grad = (2 / n) * grad_sum
    loss = loss_sum / n
    w = w - lr * grad

Tracing the arithmetic (rounded for readability):

• step 0: w = 8.000, loss = 34.5, gradient = -31.5
• step 1: w = 8.315, loss = 25.3, gradient = -26.8
• step 2: w = 8.583, loss = 18.7, gradient = -22.8
• step 3: w = 8.810, loss = 13.9, gradient = -19.3

Walking through step 0 by hand as a check: predictions are 8×1, 8×2, 8×3, 8×4 = 8, 16, 24, 32; errors are 8−12, 16−19, 24−31, 32−40 = −4, −3, −7, −8; error×x values are −4, −6, −21, −32, summing to −63; the gradient is (2/4)×(−63) = −31.5, and the loss is (16+9+49+64)/4 = 138/4 = 34.5 — exactly matching the trace. Notice the gradient's magnitude is shrinking with every step (−31.5 → −26.8 → −22.8 → −19.3) — the bowl gets flatter as you approach the bottom, so the steps naturally get gentler even though α never changes. Given enough steps, w creeps steadily toward 10.1, exactly matching the algebra answer we found by completing the square.

The Learning Rate: the Knob That Can Break Everything

Common misconception, corrected: "A bigger learning rate always makes training faster." This is false past a certain threshold — a learning rate that's too large doesn't just converge slower, it can blow up entirely. Watch what happens starting from the same w0 = 8, but with α = 0.2 instead of 0.01:

• step 0: w = 8.0 (distance from w* = 10.1 is 2.1)
• step 1: w = 14.3 (distance is 4.2)
• step 2: w = 1.7 (distance is 8.4)
• step 3: w = 26.9 (distance is 16.8)

Each step overshoots further than the last, and the distance from the true minimum exactly doubles every time. This isn't a coincidence — we can prove it. Since our loss is exactly quadratic, its gradient works out to a clean linear expression: dL/dw = 15(w − 10.1) (you can check this by substituting w = 10.1 and confirming the gradient is zero, as it must be at a minimum). Plugging this into the update rule:

w_new − w* = (w_old − w*) − α·15·(w_old − w*) = (1 − 15α)(w_old − w*)

This is a geometric progression — the kind you'll meet formally in Class 11 — with common ratio r = 1 − 15α. Each step multiplies the distance-from-optimum by r. For convergence we need |r| < 1, i.e. 0 < α < 2/15 ≈ 0.133. Our safe run used α = 0.01, giving r = 0.85 — the distance shrinks by 15% every step, matching the slow, steady convergence we traced. Our exploding run used α = 0.2, giving r = 1 − 3 = −2 — magnitude greater than 1, so the distance doubles and the sign flips every step, which is exactly the oscillating, diverging pattern above. The "15" in this formula is (2/n)·Σx_i² — it measures how sharply curved the loss bowl is. Steeper bowls need smaller learning rates; this is why real training pipelines spend real effort tuning this one number.

Scaling Up: Gradient Descent in Many Dimensions

Real models rarely have just one parameter. Let's add back the intercept we dropped earlier: predicted = w·x + b, with two parameters. The loss L(w, b) is now a surface in three dimensions (imagine the bowl extruded into a proper 3D bowl), and instead of a single derivative, we need a gradient vector — one partial derivative per parameter, found the same way as before, holding the other parameter fixed:

∂L/∂w = (2/n)·Σ(w·x_i + b − y_i)·x_i and ∂L/∂b = (2/n)·Σ(w·x_i + b − y_i)

The update rule barely changes: both parameters move opposite their own partial derivative, simultaneously, each scaled by the same learning rate. This generalizes cleanly to any number of parameters — a neural network's gradient is just a very long list of partial derivatives, one per weight, computed automatically by an algorithm called backpropagation (a separate chapter's topic), then handed to gradient descent exactly as above.

This is also where the urgency of gradient descent becomes obvious. Suppose you tried to find the best settings by brute-force search instead — testing candidate values on a grid — for a network with just 1,000 parameters (small, by modern standards; OpenAI's GPT-3, published in 2020, has 175 billion). Testing even 10 candidate values per parameter requires checking 10^1000 combinations, a number vastly larger than the roughly 10^80 atoms estimated to exist in the observable universe. Brute force isn't just slow here — it's physically impossible. Gradient descent works because at every point it only needs local slope information, not an exhaustive search, to decide which way to step.

Loss surfaces with more than one parameter also aren't always simple bowls. A surface that curves steeply in one direction and gently in another forms a "ravine," and it can have several dips (local minima) instead of one clean global minimum — genuinely non-convex, unlike our clean one-parameter quadratic. Neural network loss surfaces are almost always non-convex for exactly this reason, which is part of why the choice of optimization algorithm matters so much in practice.

Batch, Stochastic, and Mini-Batch Descent

Our worked example computed the gradient using all four data points at every single step — this is called batch gradient descent. It gives the mathematically exact gradient, but consider a payments system like India's UPI, which handles billions of transactions a month; a fraud-detection model trained on that data cannot afford to scan every single transaction before taking even one training step. Two alternatives trade exactness for speed:

• Stochastic Gradient Descent (SGD) computes the gradient using just one randomly chosen data point per step. It's extremely fast per step but the gradient estimate is noisy — it points roughly, not exactly, downhill.
• Mini-batch gradient descent is the practical middle ground used almost everywhere in real training: compute the gradient on a small random batch (say, 32 or 256 examples) at a time. It's far cheaper than full-batch, and much less noisy than single-example SGD.

Interestingly, the noise in mini-batch and stochastic gradients isn't purely a downside. Because each step's direction is only an estimate, the path can jump slightly sideways out of a shallow dip rather than settling into it — sometimes helping the optimizer escape a poor local minimum on the way to a better one, which matters precisely because real loss surfaces are non-convex, as noted above.

Momentum: Giving the Algorithm Memory

Plain gradient descent has no memory — every step is computed from scratch using only the current position, ignoring which direction it's been moving. In a ravine-shaped loss surface (steep in one direction, gentle in another, as pictured above), this causes visible zigzagging: the algorithm keeps overcorrecting back and forth across the steep direction while creeping only slowly along the gentle direction toward the actual minimum.

Momentum fixes this by accumulating a running "velocity" from past gradients, the way a ball rolling downhill keeps some of its speed instead of stopping and recalculating at every instant:

v_new = β·v_old − α·(gradient)   then   w_new = w_old + v_new

Here β (commonly around 0.9) controls how much of the previous velocity carries over. Gradients that keep pointing the same way (the gentle, productive direction) reinforce each other and build speed; gradients that keep flipping sign (the steep, oscillating direction) partly cancel out. The net effect is the smoother, faster blue path in the diagram above compared to the zigzagging red one — genuinely fewer steps to reach the minimum, not just a cosmetic difference.

A Glimpse Ahead: Adaptive Optimizers

Momentum still uses one global learning rate for every parameter. Modern training almost always uses Adam (introduced by Diederik Kingma and Jimmy Ba in 2014), which combines momentum with a separate, adaptive learning rate for every single parameter, based on running estimates of the gradient's recent mean (m) and recent squared magnitude (v):

m = β1·m + (1−β1)·g   v = β2·v + (1−β2)·g²   w_new = w_old − α·m / (√v + ε)

(typical defaults: β1 = 0.9, β2 = 0.999, ε a tiny constant to avoid dividing by zero). Parameters whose gradients have been consistently large get automatically smaller effective steps, and vice versa. You don't need to memorize this formula for Grade 10 — but you should recognize that it's built from exactly the two ideas you now understand deeply: the gradient, and momentum's running average.

Where This Shows Up in Your Exams

The core technique here — differentiate, set the result to zero (or move opposite it, iteratively) — is the same skeleton as Class 12's "Application of Derivatives" chapter, including the maxima/minima and increasing/decreasing function questions common in CBSE boards and JEE Main. Completing the square to find a quadratic's vertex, which we used to find w* = 10.1 algebraically, is a Class 10–11 staple that you've just seen double as a way to sanity-check a numerical algorithm. The geometric-progression convergence argument for the learning rate connects directly to Class 11 Sequences and Series. If you continue toward GATE-level computer science, this entire chapter is the entry point to numerical optimization and convex optimization, both examined topics; the gradient vector and partial derivatives are also foundational for GATE's linear algebra and calculus sections.

Check Your Understanding

Try these before reading the answers.

1. For L(w) = 3(w − 5)² + 2, state the minimum value and the w at which it occurs — without differentiating.
2. Differentiate that same L(w) using the power and chain rules, then evaluate the gradient at w = 7. Should gradient descent increase or decrease w from there?
3. Using the convergence condition derived above (r = 1 − 2aα for loss a(w−w*)² + c, here a = 3), will a learning rate of α = 0.5 converge or diverge? What about α = 0.1?
4. Explain, in your own words, why brute-force grid search is not merely "slow" but genuinely infeasible for a network with a million parameters.
5. A classmate says, "SGD is strictly worse than batch gradient descent because its gradient is noisy." Give one concrete reason this isn't entirely true.

Answers: (1) Minimum value 2 at w = 5, read directly off the completed-square form, exactly as we did for our loss bowl. (2) dL/dw = 6(w−5); at w=7 this is 6×2=12, positive, so gradient descent should decrease w (move opposite the positive gradient). (3) α=0.5: r = 1−2(3)(0.5) = 1−3 = −2, magnitude greater than 1 — diverges. α=0.1: r = 1−0.6 = 0.4, magnitude less than 1 — converges. (4) Because the number of combinations grows exponentially with the number of parameters (10^1000 combinations for just 1,000 parameters at 10 values each) — this dwarfs any amount of computing time available, unlike an ordinary "slow but eventually finishes" search. (5) SGD's noisy steps can jump out of shallow, unhelpful local minima that a smoother batch gradient path might settle into, and each SGD step is vastly cheaper to compute, so it can take far more steps in the same wall-clock time.

Summary

An AI model's training is a search for the parameter values that minimize a loss function. For a single parameter with squared-error loss, that loss is an algebra-solvable parabola; gradient descent is the general-purpose algorithm that finds the bottom of far more complicated (and far higher-dimensional) loss surfaces using only local slope information, via the update rule w ← w − α·(dL/dw), moving opposite the gradient because the gradient points toward steeper loss, not away from it. The learning rate α is not a minor tuning knob — push it past 2/(curvature) and the algorithm provably diverges, oscillating with ever-growing magnitude, as the geometric-progression argument shows exactly. Extending to many parameters replaces the single derivative with a gradient vector, one partial derivative per parameter, and turns brute-force search from merely slow into physically impossible. Batch, stochastic, and mini-batch descent trade gradient accuracy for computational feasibility on real-world data scales; momentum adds memory of past gradients to smooth out zigzagging in ravine-shaped surfaces; and adaptive optimizers like Adam combine momentum with a per-parameter learning rate that adjusts itself automatically. Every one of these ideas is a direct, traceable extension of the one formula you derived by hand in this chapter.