Gradient Descent: How AI Learns Step by Step

Build the smallest possible "AI model" you can imagine: one number, called a weight w, that turns an input into a prediction by simple multiplication, prediction = w × x. Suppose x is the number of practice problems a student solved before a short quiz, and you have three past students to learn from:

• Solved 1 problem → scored 2 out of 10
• Solved 2 problems → scored 4 out of 10
• Solved 3 problems → scored 8 out of 10

This is a toy dataset, kept deliberately tiny so every number in this chapter can be checked by hand — it is not claiming that scores really scale this way. The question a machine learning model has to answer is: what single value of w makes w × x the best possible predictor of the score, across all three students at once? You could guess values of w and check how wrong each guess is, but guessing does not scale to models with millions of weights. Gradient descent is the algorithm that replaces guessing with a precise, repeatable rule: look at how wrong you currently are, compute exactly which direction reduces that wrongness, and take a measured step in that direction. Do this a few thousand times and the weight converges to the value that fits the data best. This chapter builds that rule from first principles, using this exact three-point dataset as a running example so every formula stays checkable rather than abstract.

Turning "wrong" into a number: the loss function

Before you can reduce an error, you need to measure it. For a weight w, the prediction on student i is w × x_i, and the error is (w × x_i) - y_i, where y_i is the true score. Squaring the error makes it positive (so overshooting and undershooting count equally) and penalizes large errors more heavily than small ones. Averaging the squared errors over all three students gives the mean squared error, the loss function:

L(w) = (1/3) × [ (w·1 - 2)² + (w·2 - 4)² + (w·3 - 8)² ]

This looks complicated until you expand it. Each squared term is a quadratic in w, so the whole sum is quadratic in w. Expanding (w·x_i - y_i)² = x_i²w² - 2x_i y_i w + y_i² and summing over the three students:

• Sum of x_i²: 1² + 2² + 3² = 1 + 4 + 9 = 14
• Sum of x_i y_i: (1·2) + (2·4) + (3·8) = 2 + 8 + 24 = 34
• Sum of y_i²: 2² + 4² + 8² = 4 + 16 + 64 = 84

So the loss collapses to a single clean quadratic in one variable:

L(w) = (1/3)(14w² - 68w + 84) = (14/3)w² - (68/3)w + 28

This is the entire "landscape" the AI has to search. L(w) is a parabola opening upward — every quadratic with a positive leading coefficient is — so it has exactly one lowest point, and finding that point is the whole learning problem. Notice something important already: for this simple model, you could solve for the minimum directly using algebra (set the derivative to zero, which we do below). Real neural networks have millions of weights and a loss surface with no such clean closed-form solution, which is exactly why they need an iterative, step-by-step search instead. This tiny example is a laboratory for understanding that search before you meet the version with no algebraic shortcut.

The derivative tells you which way is downhill

You may not yet have formally covered derivatives in your regular Class 10 mathematics — that is fine, because the only two rules you need are simple ones you can verify from the definition of slope: the derivative of w² is 2w, the derivative of w is 1, and the derivative of a constant is 0. Applying these term by term to L(w) = (14/3)w² - (68/3)w + 28:

L'(w) = (14/3)·2w - (68/3)·1 + 0 = (28/3)w - 68/3 = (28w - 68)/3

L'(w) is the slope of the loss curve at the point w. If L'(w) is negative, the curve is falling as w increases — so increasing w reduces the loss. If L'(w) is positive, the curve is rising as w increases — so you should decrease w instead. Either way, moving in the direction opposite the sign of the derivative reduces the loss, at least for a small enough move. That single observation is the entire idea behind gradient descent: repeatedly nudge w in the direction opposite its own slope.

Why the opposite-of-slope direction actually works

This deserves a real justification, not just intuition. Near any point w, a small step Δw changes the loss by approximately L(w + Δw) − L(w) ≈ L'(w) · Δw — this is the first-order (linear) approximation of a function near a point, and for a quadratic like ours it becomes an exact identity once you also add the constant second-order term, but the linear term is what controls the direction of change. Now choose the step to be Δw = -η · L'(w) for some small positive number η (the learning rate). Substituting:

L(w + Δw) - L(w) ≈ L'(w) · (-η · L'(w)) = -η · [L'(w)]²

Since [L'(w)]² is a square, it is never negative, and η is positive, so the right-hand side is never positive. The loss is guaranteed not to increase from this step (and strictly decreases whenever the slope is not already zero) — provided the step is small enough for the linear approximation to hold. This is not a heuristic; it is a direct algebraic consequence of choosing the step to move opposite the slope. It also tells you precisely what can go wrong: if η is too large, the step is no longer "small," the approximation breaks down, and the guarantee evaporates. You will see exactly this failure happen with real numbers later in this chapter.

The gradient descent update rule

Putting the last two sections together gives the formal rule used to train essentially every machine learning model in use today:

w ← w - η · L'(w)

Read this as: "replace w with w minus the learning rate times the slope of the loss at the current w." You start from some initial guess w₀ (often 0, or a small random number), then repeatedly apply this update. Each application is called a step or an iteration. The learning rate η is a number you choose before training begins; it controls how big each step is, not which direction it goes — the direction is decided entirely by the sign of L'(w).

Worked example: watching the weight converge

Using L'(w) = (28w - 68)/3, start at w₀ = 0 with a learning rate of η = 0.05. The Python loop below implements the rule exactly as written above, then prints the weight and loss at each step before updating:

def gradient(w):
    return (28 * w - 68) / 3

def loss(w):
    return (14 * w**2 - 68 * w + 84) / 3

w = 0.0
eta = 0.05
for step in range(6):
    print(step, round(w, 4), round(loss(w), 4))
    w = w - eta * gradient(w)

Tracing this by hand, step by step: at w₀ = 0, L'(0) = -68/3 ≈ -22.667, so w₁ = 0 - 0.05 × (-22.667) = 1.1333. At w₁ = 1.1333, L'(w₁) ≈ -12.089, giving w₂ = 1.1333 + 0.05 × 12.089 = 1.7378. Continuing this exact arithmetic produces w₃ = 2.0601, w₄ = 2.2321, and w₅ = 2.3238. The printed output is:

0 0.0 28.0
1 1.1333 8.3052
2 1.7378 2.7031
3 2.0601 1.1096
4 2.2321 0.6564
5 2.3238 0.5274

The weight is climbing steadily toward the true minimum, and the loss is shrinking toward it every single step — from 28 down to 0.527 in five iterations. Where is it heading? Set L'(w) = 0 directly: (28w - 68)/3 = 0 ⇒ w = 68/28 = 17/7 ≈ 2.4286. This is w*, the exact minimum, and you can see the sequence w₀, w₁, …, w₅ creeping toward exactly this value. At the minimum itself, the loss is L(17/7) = 10/21 ≈ 0.4762 — the lowest value the parabola ever reaches.

The diagram below plots this exact loss curve and marks the path the weight actually took, using the same numbers computed above — every dot sits precisely on the curve, at the pixel position its own w and L(w) value produce.

Every orange dot lies exactly on the blue curve, because each one is a point on L(w) — the weight is literally rolling downhill along the surface of its own loss function, taking smaller and smaller steps as the slope flattens near the bottom. That flattening is not a coincidence: as w approaches w*, L'(w) approaches zero, so η · L'(w) (the step size) shrinks automatically even though η itself stays fixed.

Common misconception: the gradient does not know where the minimum is

A very natural but incorrect belief is that gradient descent somehow "sees" the whole parabola and calculates the shortest path to the bottom. It does not. At every step, the algorithm only ever evaluates L'(w) at the current w — a purely local measurement, like feeling the steepness of the ground under your feet in thick fog. It has no memory of the curve's overall shape and no preview of what lies further along. It works for this simple parabola only because a parabola has a special property: local information (the slope right here) always points reliably toward the single global minimum, no matter where you stand. As you will see in the section on convexity below, this reliability is not automatic for more complicated loss surfaces, which is precisely why it deserves to be called out as a property of this shape rather than a general truth about gradient descent.

When the learning rate is too large: a precise account of divergence

The learning rate is not a minor tuning knob — get it wrong and the algorithm actively moves away from the minimum. To see exactly why, rewrite the loss in "vertex form" centered on the minimum: L(w) = a(w - w*)² + L_min, where here a = 14/3 (the coefficient you get by matching 2a = 28/3 to the slope of L') and w* = 17/7. Differentiating this form directly gives L'(w) = 2a(w - w*). Substitute this into the update rule:

w_new - w* = (w - η·L'(w)) - w* = (w - w*) - η·2a(w - w*) = (1 - 2aη)·(w - w*)

Call r = 1 - 2aη. This equation says the distance from the minimum gets multiplied by r at every single step. If |r| < 1, the distance shrinks every time and the algorithm converges — that is exactly what happened above, since η = 0.05 gives r = 1 - (28/3)(0.05) = 8/15 ≈ 0.533. But if |r| > 1, the distance grows every step, without bound.

Try η = 0.3 instead, starting again from w₀ = 0. Here r = 1 - (28/3)(0.3) = 1 - 2.8 = -1.8, so |r| = 1.8 > 1: this learning rate is already predicted to diverge before computing a single step. Checking directly: L'(0) = -68/3, so w₁ = 0 - 0.3 × (-68/3) = 6.8. The starting distance from the minimum was |w₀ - w*| = |0 - 2.4286| = 2.4286. The new distance is |w₁ - w*| = |6.8 - 2.4286| = 4.3714. The ratio 4.3714 / 2.4286 = 1.8 — exactly |r|, as the formula predicted. In words: the single step overshot the minimum by 1.8 times the starting distance — the error grew instead of shrinking, and it does not stop there. Taking a second step from w₁ = 6.8: L'(6.8) = (28 × 6.8 - 68)/3 = 40.8, giving w₂ = 6.8 - 0.3 × 40.8 = -5.44, now 7.87 units from the minimum — worse again, by the same factor of 1.8. This is textbook divergence: the weight oscillates from one side of the minimum to the other, growing further away each time.

The formula r = 1 - 2aη also tells you exactly how to choose a safe learning rate for this problem: convergence requires |1 - 2aη| < 1, which simplifies to 0 < η < 1/a = 3/14 ≈ 0.2143. Our working example used η = 0.05, comfortably inside this range; the divergent example used η = 0.3, well outside it. There is even a single "perfect" learning rate that lands exactly on the minimum in one step: setting r = 0 gives η = 1/(2a) = 3/28 ≈ 0.1071. This exact-in-one-step behaviour is a special feature of quadratic loss functions and will not happen for the more complex, non-quadratic losses used by real neural networks — but the general lesson, that there is a boundary beyond which larger steps make things worse, not better, carries over exactly.

Why convexity matters

Everything above worked cleanly because L(w) is convex: a parabola opening upward, with one minimum and no other flat or dipping regions to get trapped in. For a convex loss, the local slope at any point always points toward the single global minimum, which is exactly why "follow the negative slope" is guaranteed to work, given a suitable learning rate. Linear regression with squared error, like the toy example here, is always convex in its weights, which is why it can even be solved exactly with algebra instead of iteration.

Deep neural networks are a different story. Their loss surfaces, as functions of millions of weights, are generally non-convex: full of many local dips, flat plateaus, and saddle points, not just one clean bowl. Gradient descent on such a surface can still make steady local progress, but it has no guarantee of finding the single best solution anywhere on the surface — it may settle into a local dip that is good enough for the task, which turns out to work remarkably well in practice for systems like handwriting-recognition apps that convert photographed notes into digital text, or translation tools such as Google Translate. The core mechanics — measure the local slope, step opposite it, scaled by a learning rate — are identical to what you traced by hand in this chapter; what changes at scale is the shape of the surface being descended and the number of weights being adjusted simultaneously.

From one weight to many: the gradient vector

Real models rarely have just one weight. A model with weights w₁, w₂, …, w_n has a loss L(w₁, w₂, …, w_n) that depends on all of them together. The natural generalization of "the derivative" here is the partial derivative ∂L/∂w_i: how the loss changes if you nudge only w_i while holding every other weight fixed. Collecting all of these partial derivatives into one list gives the gradient, written ∇L = (∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂w_n) — a vector, not a single number. Every weight is then updated simultaneously using its own partial derivative: w_i ← w_i - η · ∂L/∂w_i, for every i at once. This is exactly the update rule you derived above, applied component by component; only the bookkeeping changes, not the underlying idea. This is also the origin of the algorithm's full name — "gradient" descent, not just "derivative" descent — and the reason the technique scales from the one-weight toy model in this chapter to networks with billions of weights: it is the same rule, run once per weight, at every step.

Where this fits in your exam preparation

For CBSE Boards, the calculus tools used here — differentiating polynomials, finding where a derivative equals zero to locate a minimum, and reasoning about the sign of a derivative — are the same techniques examined under Applications of Derivatives in the Class 12 syllabus; working through this chapter's derivations by hand is genuine practice for that unit, not a detour from it. For IIT-JEE and BITSAT, the convergence analysis above — tracking a quantity that gets multiplied by a fixed ratio r at every step, and asking when the resulting sequence converges or diverges — is precisely the reasoning used in Sequences and Series problems about geometric-type recurrences, and questions on maxima/minima of polynomial functions draw on the same derivative-equals-zero technique used to find w* here. For students continuing toward GATE-level computer science or a future AI/ML specialization, gradient descent is foundational vocabulary: this chapter's one-weight derivation is the exact mechanism, just without the extra bookkeeping, used inside every deep learning framework's training loop. Olympiad-style questions on iterative sequences and fixed points also draw on the same |r| < 1 convergence condition derived above. (Note for readers checking older material: the KVPY scholarship exam was discontinued in 2021 and replaced by INSPIRE-SHE as the relevant national-level science aptitude exam.)

Check your understanding

1. Compute one step by hand. Using L'(w) = (28w - 68)/3 and η = 0.05, start at w = 1 instead of 0. What is w after one update? (Work it out before checking: L'(1) = (28 - 68)/3 = -40/3 ≈ -13.333, so w_new = 1 - 0.05 × (-13.333) = 1 + 0.6667 = 1.6667.)
2. Find the one-step-exact learning rate. Using a = 14/3, solve 1 - 2aη = 0 for η and confirm it equals 3/28. Then verify by hand that starting from w₀ = 0 with this η lands exactly on w* = 17/7 after one step.
3. Predict divergence without simulating. Without computing any weight values, decide whether η = 0.25 converges or diverges for this loss function, using only the condition 0 < η < 1/a. (Since 1/a = 3/14 ≈ 0.2143 and 0.25 > 0.2143, this learning rate diverges.)
4. Conceptual check. Explain, in your own words, why the gradient descent update rule contains no information about where the global minimum is located, even though it reliably finds it for this particular loss function.

Summary

• A loss function L(w) measures how wrong a model's predictions are for a given weight; gradient descent finds the weight that minimizes it.
• The derivative L'(w) gives the local slope; moving opposite its sign, by an amount scaled by the learning rate η, is guaranteed (for a small enough step) to reduce the loss — a direct consequence of the linear (first-order) approximation of L near the current point.
• The update rule is w ← w - η·L'(w), applied repeatedly until the weight stops changing meaningfully.
• Near the minimum of a quadratic loss, the distance to the minimum shrinks by a fixed ratio r = 1 - 2aη at every step; convergence requires |r| < 1, i.e. 0 < η < 1/a. Too large a learning rate makes |r| > 1 and the algorithm diverges, oscillating further from the minimum with every step.
• Convexity (a single bowl-shaped loss, no other dips) is what guarantees that following the local slope leads to the true global minimum; deep neural networks generally lack this guarantee, yet the same slope-following update rule remains their core training mechanism.
• With many weights, the single derivative generalizes to the gradient vector ∇L, one partial derivative per weight, all updated simultaneously — the same rule you derived here, run once per parameter.