Gradient Descent: Optimization Algorithm Explained

Suppose a friend asks you to guess a number between 0 and 20 that they are thinking of. Every time you guess, they don't tell you the number — they only tell you "colder" (you moved away) or "warmer" (you moved closer), and roughly by how much. You would not guess randomly forever. You would guess something, check whether you got warmer or colder, and adjust your next guess in the direction that made you warmer. Guess after guess, you would close in on the answer without ever seeing it directly.

This is almost exactly what a machine learning algorithm called gradient descent does — except instead of guessing a hidden number, it is guessing the best value of a parameter inside a prediction model, and instead of "warmer/colder" from a friend, it uses the slope of a mathematical function to know which direction to move and by how much. Gradient descent is the workhorse algorithm that trains almost every machine learning model you will ever use — from a simple line-of-best-fit to the neural networks behind voice assistants and photo-tagging apps. In this chapter you will learn exactly how it works, trace it by hand on real numbers, write it in Python, and see precisely where it can go wrong.

Turning "find the best guess" into a math problem

Every prediction model makes mistakes. If a model predicts a value ŷ (pronounced "y-hat") and the true value is y, the error is y − ŷ. To judge how good a model's parameters are overall, we combine all its errors into a single number called a cost function (also called a loss function) — a function that is small when the model is doing well and large when it is doing badly. The single job of training a model is: find the parameter values that make the cost function as small as possible. That is an optimization problem, and gradient descent is the algorithm we use to solve it.

Let's strip away the machine learning vocabulary for a moment and work with the simplest possible version of this problem: a cost function of just one variable,

cost(x) = (x - 4)^2

Imagine this represents the total squared error of some simple model whose only tunable knob is x. By inspection you can see the smallest possible value of cost(x) is 0, reached when x = 4 — because squaring a real number can never be negative, and it equals zero only when the number being squared is zero. But a computer training a real model usually cannot "inspect" the function like this; the cost function might depend on millions of parameters and there is no way to eyeball the answer. It needs a systematic, repeatable procedure. That procedure is gradient descent.

The slope tells you which way is downhill

You already know from coordinate geometry that the slope of a straight line is rise over run: how much y changes for a small change in x. A curve does not have one fixed slope everywhere — but at any single point on a curve, you can still ask "if I nudge x slightly, does cost(x) go up or down, and how fast?" That local steepness at a point is called the gradient at that point. (For a function of one variable, "gradient" and "slope of the tangent" mean the same thing; for functions of several variables, the gradient is simply the collection of slopes with respect to each variable separately, which we will use later.)

For cost(x) = (x − 4)^2, the slope at any point x works out to 2(x − 4) — a formula you can verify numerically: at x = 6, nudging x up by a tiny amount increases cost quickly (steep, positive slope, curve rising to the right of the minimum); at x = 2, nudging x up actually decreases cost (negative slope, curve falling toward the minimum). This is the key fact gradient descent relies on:

• If the slope at your current x is positive, the function is rising as x increases — so the minimum is to your left. You should decrease x.
• If the slope is negative, the function is falling as x increases — so the minimum is to your right. You should increase x.

Notice the pattern: you always want to move opposite to the sign of the slope. That single observation is the entire idea of gradient descent, written as one line of algebra:

new_x = old_x - learning_rate * slope_at(old_x)

Subtracting the slope (scaled by a small positive number called the learning rate) automatically moves you downhill, regardless of which side of the minimum you're standing on. This update rule is applied over and over — it is an iterative algorithm, not a one-shot calculation.

Tracing it by hand: five real steps

Let's run this by hand on cost(x) = (x − 4)^2, whose slope is 2(x − 4). Start at x = 0 (a deliberately bad guess) with learning rate 0.1.

Step 0: x = 0.0000   slope = 2(0 - 4)      = -8.0000   cost = 16.0000
Step 1: x = 0.8000   slope = 2(0.8 - 4)    = -6.4000   cost = 10.2400
Step 2: x = 1.4400   slope = 2(1.44 - 4)   = -5.1200   cost = 6.5536
Step 3: x = 1.9520   slope = 2(1.952 - 4)  = -4.0960   cost = 4.1943
Step 4: x = 2.3616   slope = 2(2.3616-4)   = -3.2768   cost = 2.6844
Step 5: x = 2.6893   slope = 2(2.6893-4)   = -2.6214   cost = 1.7180

Check the arithmetic for step 1 yourself: slope at x = 0 is 2(0 − 4) = −8. New x = 0 − 0.1 × (−8) = 0 + 0.8 = 0.8. Because the slope was negative, subtracting a negative number pushed x up, toward 4 — exactly the rule from the previous section. At x = 0.8, the new slope is 2(0.8 − 4) = −6.4, still negative but smaller in magnitude because we are closer to the minimum, so the next step (+0.64) is smaller than the first (+0.8). This is a general feature of gradient descent on a bowl-shaped function: steps automatically shrink as you approach the minimum, because the slope itself shrinks near a flat bottom. The diagram below plots exactly these six points on the curve.

Notice how the red points bunch closer together as they approach the green minimum marker — that visual shrinking of step size is the signature of gradient descent converging correctly.

Writing it as code

Translating the update rule into Python is direct — this is the entire algorithm, with no shortcuts or hidden library calls:

def cost(x):
    return (x - 4) ** 2

def slope(x):
    return 2 * (x - 4)

x = 0.0
learning_rate = 0.1

for step in range(6):
    print(f"step {step}: x = {x:.4f}, cost = {cost(x):.4f}")
    x = x - learning_rate * slope(x)

Running this prints exactly the six rows from the hand-traced table above: step 0: x = 0.0000, cost = 16.0000 through step 5: x = 2.6893, cost = 1.7180. If you keep the loop running for, say, 50 steps instead of 6, x keeps creeping toward 4.0000 and cost keeps shrinking toward 0, but mathematically it never exactly reaches 4 in finite steps — each step only closes part of the remaining gap (here, exactly 20% of it, since the update always removes 2 × learning_rate = 0.2 of the current distance). In real training code, we don't run forever; we stop once the change in cost between steps becomes smaller than some tiny tolerance, or after a fixed number of iterations called epochs.

The general rule, with more than one parameter

Real prediction models almost never have just one tunable number. Consider predicting the price of a prepaid mobile data plan from the amount of data it offers. A simple linear model is price = m × data_GB + c, where m (the slope) and c (the intercept) are the two parameters we must learn from past pricing data. Now the cost function depends on two variables, m and c, so it no longer looks like the single curve above — it looks like a bowl-shaped surface sitting over the (m, c) plane, and gradient descent has to walk downhill on that surface in two directions at once.

The idea does not change — only the bookkeeping does. The gradient is now a slope for each parameter separately: how cost changes if you nudge m alone, and how cost changes if you nudge c alone. The update rule becomes two update rules running side by side:

m = m - learning_rate * (slope of cost with respect to m)
c = c - learning_rate * (slope of cost with respect to c)

For the standard cost function used in linear regression — mean squared error, MSE = (1/n) × sum of (y_actual - y_predicted)^2 over all n data points — those two slopes have known formulas:

slope w.r.t. m = -(2/n) * sum( x_i * (y_i - (m*x_i + c)) )
slope w.r.t. c = -(2/n) * sum(       y_i - (m*x_i + c)   )

Take a tiny dataset of four data plans (data in GB, price in rupees) that happens to follow a perfectly straight line: (1, 19), (2, 29), (3, 39), (4, 49) — each extra GB costs exactly ₹10 more, plus a fixed ₹9 base charge, so the true best-fit line is m = 10, c = 9. Starting both parameters at zero, the first update step computes the errors as simply the prices themselves (since the prediction is 0 for everything): errors are 19, 29, 39, 49. Plugging into the formulas: slope w.r.t. m = -(2/4) × (1×19 + 2×29 + 3×39 + 4×49) = -(0.5) × 390 = -195, and slope w.r.t. c = -(2/4) × (19+29+39+49) = -(0.5) × 136 = -68. With a learning rate of 0.01, the very first update moves m from 0 to 0 - 0.01 × (-195) = 1.95, and c from 0 to 0 - 0.01 × (-68) = 0.68 — both nudged in the right direction, toward the true values 10 and 9. Here is the full loop as runnable code:

data = [(1, 19), (2, 29), (3, 39), (4, 49)]  # (GB, price in Rs)

m, c = 0.0, 0.0
learning_rate = 0.01
n = len(data)

for epoch in range(2000):
    sum_m, sum_c = 0.0, 0.0
    for x, y in data:
        prediction = m * x + c
        error = y - prediction
        sum_m += x * error
        sum_c += error
    grad_m = -(2 / n) * sum_m
    grad_c = -(2 / n) * sum_c
    m = m - learning_rate * grad_m
    c = c - learning_rate * grad_c

print(f"m = {m:.2f}, c = {c:.2f}")

Run this yourself: as the epochs progress, m climbs toward 10.00 and c climbs toward 9.00, with the total error shrinking toward zero, because this particular dataset is perfectly linear and reachable by the model. This is precisely how the "line of best fit" you may compute directly using formulas in statistics can also be learned iteratively — and iterative learning is the approach that scales to models with millions of parameters, where no direct formula exists.

Why square the error, and why subtract the gradient?

Two design choices are worth pinning down explicitly, because they are often taken for granted. First, we square the error rather than just summing (y − ŷ) directly, for two reasons: squaring makes every error positive (so a +5 error and a −5 error don't cancel out and hide each other), and it penalizes large errors much more heavily than small ones (an error of 10 contributes 100 to the cost, not 10), which pushes the model harder to fix its worst mistakes. Second, we always subtract the gradient, never add it, because the gradient by definition points in the direction of steepest increase of the cost. Moving opposite to it is the only way to guarantee you are moving toward lower cost, regardless of which side of the minimum you currently sit on — that single minus sign is what makes the algorithm "descend" rather than "ascend."

Common misconception: "a bigger learning rate always reaches the answer faster"

This is false, and getting it wrong is the single most common bug when people implement gradient descent for the first time. The learning rate controls how big a jump you take in the downhill direction — but a jump that is too large can overshoot the minimum entirely and land on the far side of the bowl, at a point where the cost is worse than where you started. Watch what happens on the same cost(x) = (x − 4)^2 function, starting at x = 0, but with learning rate 1.5 instead of 0.1:

Step 0: x = 0.0    cost = 16.0
Step 1: x = 12.0    cost = 64.0
Step 2: x = -12.0    cost = 256.0
Step 3: x = 36.0    cost = 1024.0

Check step 1: slope at x=0 is -8, so new_x = 0 − 1.5×(−8) = 12. But 12 is even farther from the true minimum (4) than 0 was, and the cost has quadrupled instead of shrinking. Each subsequent step overshoots further in the opposite direction, and the cost explodes — this is called divergence, and it is a genuine failure mode, not a rare edge case. On the other hand, a learning rate that is too small, such as 0.001, is always numerically safe but converges extremely slowly: matching the progress that 0.1 achieved in just 5 steps above would take a learning rate of 0.001 more than 500 steps, because each update chips away only a tiny fraction of the remaining distance. Correct practice is to pick a learning rate small enough to avoid overshoot but large enough to make real progress — often found by trying a few values (like 0.1, 0.01, 0.001) and watching whether the cost decreases smoothly or explodes.

Local minima versus the global minimum

Everything above worked cleanly because (x − 4)^2 is a single, symmetric bowl — mathematicians call this a convex function, and it is a mathematical fact that a convex cost function has exactly one minimum, so gradient descent starting from any point will always reach it (given a suitable learning rate). This is also true for the mean-squared-error cost function of linear regression used above — it is convex in m and c, so gradient descent is guaranteed to find the single best-fit line no matter where m and c are initialized.

Not every cost function is this well-behaved. Many real models — especially deep neural networks, which have thousands or millions of parameters — have cost surfaces with several valleys of different depths. Gradient descent only ever looks at the local slope under its current position; it has no way of "seeing" whether a deeper valley exists somewhere else. If it walks into a shallow valley first, it will happily stop there, believing it has found the best answer, even though a better (lower-cost) valley exists elsewhere on the surface. The shallow valley is called a local minimum; the truly best valley is the global minimum.

In the diagram, a run starting at start A slides down into the shallow valley and stops there — a local minimum. A run starting at start B slides down the other slope and reaches the genuinely deepest valley — the global minimum. Both runs used the identical algorithm; only the starting point differed. In practice, engineers deal with this by running gradient descent multiple times from different random starting points, by adding controlled randomness to the steps (a variant called stochastic gradient descent), or by using more advanced update rules that carry "momentum" through shallow dips instead of stopping the instant the local slope becomes zero. For a Grade 9 CBSE learner, the essential takeaway is narrower and firmer: gradient descent finding a minimum is not the same guarantee as gradient descent finding the minimum, and whether that distinction matters depends entirely on whether the cost surface is convex (one valley, always safe) or not (possibly many valleys, starting point matters).

Check your understanding

1. For cost(x) = (x − 10)^2, the slope at any x is 2(x − 10). Starting at x = 2 with learning rate 0.25, compute x after exactly two update steps, showing each slope calculation.
2. A classmate sets the learning rate to 3.0 while training a simple model and finds that the cost increases every single epoch instead of decreasing. Diagnose the problem in one sentence and suggest a fix.
3. Explain, using the words "convex" and "local minimum," why gradient descent is completely safe for training a single-feature linear regression model but not necessarily safe for training a large neural network.
4. A piece of code updates a parameter with the line x = x + learning_rate * slope instead of x = x - learning_rate * slope. Starting at x = 0 for cost(x) = (x − 4)^2, trace two steps with learning rate 0.1 and describe what goes wrong.
5. Why do we use an iterative algorithm like gradient descent to find the best-fit line for the data-plan example, when solving two linear equations directly (as you do in Class 9 algebra for two unknowns) could, in principle, find m and c in one shot for a small dataset?

Answers. (1) Step 1: slope = 2(2−10) = −16, x = 2 − 0.25×(−16) = 6. Step 2: slope = 2(6−10) = −8, x = 6 − 0.25×(−8) = 8. (2) The learning rate is too large, causing the updates to overshoot the minimum and diverge; the fix is to reduce the learning rate (try smaller values like 0.1 or 0.01). (3) Linear regression's mean-squared-error cost is convex — a single bowl-shaped surface with exactly one minimum — so gradient descent from any starting point reaches that one true minimum; a neural network's cost surface is not convex and can have many local minima, so where training starts can determine which (possibly worse) minimum it settles into. (4) Slope at x=0 is −8; buggy update gives x = 0 + 0.1×(−8) = −0.8 (moved away from 4, not toward it). Next slope = 2(−0.8−4) = −9.6; x = −0.8 + 0.1×(−9.6) = −1.76, even farther away — the sign error makes the algorithm climb uphill and diverge instead of descend. (5) Direct algebraic solving works for a handful of parameters on a small, well-behaved dataset, but it does not scale: real models can have millions of parameters and cost functions with no closed algebraic solution at all (as with most neural networks), so an iterative numerical method that only needs local slope information at each step is the only practical approach.

Summary

• A cost function measures how wrong a model's predictions are; training a model means finding the parameter values that make this function as small as possible.
• The gradient (slope) at a point tells you the direction of steepest increase; gradient descent moves the opposite way, using the rule new_value = old_value − learning_rate × gradient, repeated over many iterations (epochs) until the cost stops shrinking meaningfully.
• Step sizes shrink automatically near the minimum because the slope itself flattens out there — this is why gradient descent converges rather than overshooting on a well-behaved bowl-shaped function.
• The learning rate is a critical tuning choice: too large causes overshoot and divergence (cost explodes); too small causes safe but very slow convergence.
• For models with several parameters (like m and c in a line), the gradient is a separate slope for each parameter, and every parameter is updated simultaneously using its own slope.
• Gradient descent is guaranteed to find the true best answer only when the cost surface is convex (one valley); on non-convex surfaces, such as those in large neural networks, it can get trapped in a local minimum that is worse than the true global minimum, and the starting point matters.