Backpropagation: How Neural Networks Learn

A Network That's Wrong — Now What?

Say you've built a tiny neural network — a few neurons, some weights, a bias or two — and you feed it an input. It produces a prediction. The prediction is wrong. Not wildly wrong, just off by a few units. You know, from earlier work with these networks, that the weights are the knobs that control the output. So the question becomes painfully specific: which knob do you turn, in which direction, and by how much?

This is harder than it sounds, because in even a small network, every weight influences the output through a chain of other neurons. Turning up one weight might push the prediction closer to correct — or it might push it further away, or barely move it at all, depending on everything downstream of it. Guessing is not an option once a network has hundreds or millions of weights. You need an exact, calculable answer to "how much is this specific weight responsible for the error?" for every single weight, cheaply enough to do it after every training example.

Imagine a 4×100m relay team that misses its qualifying time by six-tenths of a second. The coach doesn't just tell everyone "run faster." She looks at each runner's split against their expected split and works out, leg by leg, how many hundredths of a second each runner actually cost the team — because the final time is the combined result of four sequential contributions, and improving the team means correctly assigning blame to each leg before deciding who trains harder. Backpropagation is the neural network's version of that split-time analysis: a precise, mechanical way to assign "blame" for the final error to every weight in the network, one layer at a time, moving backward from the output to the input.

Quick Recap: The Forward Pass

Before we can go backward, we need to fix exactly what "forward" looks like, because backpropagation reuses every quantity the forward pass computes. We'll use the smallest network that still has everything backpropagation needs to demonstrate: one input, one hidden neuron with a ReLU activation, and one output neuron with a linear (identity) activation.

• z1 = w1·x + b1 — the hidden neuron's weighted sum: input x scaled by weight w1, plus bias b1.
• h = ReLU(z1) = max(0, z1) — the hidden neuron's activation. ReLU passes positive values through unchanged and clips negative values to zero.
• z2 = w2·h + b2 — the output neuron's weighted sum, built from the hidden activation.
• o = z2 — the output neuron uses a linear activation, so its output equals its weighted sum directly.
• L = (o - y)² — the loss function, squared error against the target value y. Squaring guarantees the loss is always positive and penalizes large misses more heavily than small ones.

Four numbers control this network's behavior: w1, b1, w2, b2. (x and y are given by the training example, not trained.) Backpropagation's entire job is to compute, for a given training example, exactly how much the loss L would change if we nudged each of w1, b1, w2, b2 by a tiny amount. We write this as dL/dw1, read "how L changes per unit change in w1." Formally these are called partial derivatives and are written with the symbol ∂ instead of d, but the idea — "sensitivity of the loss to a tiny nudge in this one variable" — is all you need, and that's the notation we'll use throughout.

The Chain Rule: Multiplying Sensitivities Along a Path

Here's the key algebraic fact that makes backpropagation possible, and we can derive it with nothing more than expanding a square. Suppose a quantity o depends on y through L = (o - y)², and we nudge o by a tiny amount ε. Then:

(o - y + ε)² = (o - y)² + 2(o - y)·ε + ε²

For a tiny ε (say 0.001), the ε² term (0.000001) is negligible compared to the 2(o-y)·ε term. So the loss changes by approximately 2(o-y)·ε — meaning for every unit o increases, L increases by about 2(o-y). That's the local sensitivity dL/do = 2(o-y), and we got it from ordinary algebra, not calculus rules memorized from a textbook.

Now here's the chain rule itself, and it's just common sense once you see it: if w1 affects z1, and z1 affects h, and h affects z2, and z2 affects o, and o affects L — then nudging w1 ripples through every link in that chain, and the total effect on L is the product of each link's local effect:

dL/dw1 = (dL/do) × (do/dz2) × (dz2/dh) × (dh/dz1) × (dz1/dw1)

Think of it like currency conversion: rupees to dollars to euros. If ₹1 buys $0.012, and $1 buys €0.92, then ₹1 buys 0.012 × 0.92 euros — you multiply the exchange rates along the chain. Sensitivities chain the same way. Each factor above is something we can compute using only the local formula at that one step — nobody needs to understand the whole five-step chain to compute their own link.

Why Go Backward Instead of Testing Each Weight?

You might ask: why not just nudge w1 by a tiny amount, rerun the whole forward pass, see how much the loss changed, and repeat for every weight? This works in principle — it's called numerical differentiation — but it is ruinously expensive. A network with a million weights would need a million full forward passes just to find the gradient for one training example, and you need gradients for every example, every step of training.

Backpropagation avoids this entirely. Notice in the chain rule above that dL/do is needed to compute the gradient for every weight upstream of it. Rather than recomputing it separately for w1, b1, w2, and b2, we compute it exactly once — at the output — and reuse it as we walk backward. One forward pass to compute the values, one backward pass to compute every single gradient by reusing shared intermediate results. That reuse, not just "going backward," is the actual reason the algorithm scales to networks with millions of parameters.

Worked Example: Every Gradient, By Hand

Let's put real numbers through this. Our network: w1 = 3, b1 = 1, w2 = 2, b2 = -1. One training example: input x = 2, target y = 10.

Forward pass — compute left to right:

• z1 = 3×2 + 1 = 7
• h = ReLU(7) = 7 (positive, so ReLU passes it through unchanged)
• z2 = 2×7 + (-1) = 13
• o = z2 = 13
• L = (13 - 10)² = 3² = 9

The network predicted 13; the target was 10; the loss is 9. Now we walk backward, computing one local sensitivity at a time and multiplying as we go — exactly the chain rule from the previous section, done numerically.

Step 1 — how sensitive is the loss to the output? We derived dL/do = 2(o - y) above. Here: dL/do = 2×(13-10) = 6. Since o is too high and this number is positive, it tells us "increasing o increases the loss" — exactly what we'd expect, since our prediction already overshot the target.

Step 2 — pass through the linear output activation. Since o = z2 exactly, nudging z2 by ε nudges o by exactly ε, so do/dz2 = 1. Chaining: dL/dz2 = dL/do × do/dz2 = 6 × 1 = 6.

Step 3 — the two local sensitivities at the output neuron's weighted sum. Since z2 = w2·h + b2, nudging w2 by ε changes z2 by ε·h (expand (w2+ε)h + b2 = w2h + b2 + εh), so dz2/dw2 = h = 7. Nudging b2 by ε changes z2 by exactly ε, so dz2/db2 = 1. Chaining:

• dL/dw2 = dL/dz2 × dz2/dw2 = 6 × 7 = 42
• dL/db2 = dL/dz2 × dz2/db2 = 6 × 1 = 6

These two are done — w2 and b2 only feed into z2, nothing further upstream needs them. But we still need to continue the chain leftward, because w1 and b1 live further back, on the other side of the hidden neuron.

Step 4 — sensitivity to the hidden activation. By the same expansion as Step 3, nudging h by ε changes z2 by ε·w2, so dz2/dh = w2 = 2. Chaining: dL/dh = dL/dz2 × dz2/dh = 6 × 2 = 12.

Step 5 — through the ReLU. ReLU's local sensitivity depends on which "side" it's on. For z1 > 0, ReLU behaves like h = z1, so nudging z1 nudges h by the same amount — slope 1. For z1 < 0, ReLU outputs a flat zero no matter how z1 wiggles nearby — slope 0. Here z1 = 7 > 0, so dh/dz1 = 1. Chaining: dL/dz1 = dL/dh × dh/dz1 = 12 × 1 = 12.

Step 6 — the two local sensitivities at the hidden neuron's weighted sum. Exactly as in Step 3 but one layer back: dz1/dw1 = x = 2 and dz1/db1 = 1. Chaining:

• dL/dw1 = dL/dz1 × dz1/dw1 = 12 × 2 = 24
• dL/db1 = dL/dz1 × dz1/db1 = 12 × 1 = 12

Every gradient is now known: dL/dw1 = 24, dL/db1 = 12, dL/dw2 = 42, dL/db2 = 6. Notice we never recomputed dL/do or dL/dz2 — each was computed once and reused for both the weight and bias that depend on it, exactly the efficiency argument from the previous section.

Seeing the Whole Pass as One Diagram

The diagram below shows both passes on the same computation graph: the forward pass in blue, left to right, producing the values we just computed; the backward pass in red, right to left, producing the gradients. The dashed taps show where a backward value branches off to produce a weight/bias gradient, while the main red line continues further left, carrying the reused sensitivity to the next layer.

From Gradients to Learning: The Update Step

A gradient alone doesn't change anything — it's just a measurement. To actually learn, we use a separate, much simpler rule called gradient descent: nudge every weight a small step in the opposite direction of its gradient, since the gradient points toward increasing loss and we want to decrease it.

w_new = w_old - learning_rate × dL/dw

The learning_rate is a small number we choose (say 0.01) that controls the step size. Applying it to our four gradients:

• w1 = 3 - 0.01×24 = 2.76
• b1 = 1 - 0.01×12 = 0.88
• w2 = 2 - 0.01×42 = 1.58
• b2 = -1 - 0.01×6 = -1.06

Let's check that this actually helped by running the forward pass again with the new weights: z1 = 2.76×2 + 0.88 = 6.4, h = 6.4, z2 = 1.58×6.4 - 1.06 = 9.052, so the new prediction is o = 9.052 against target 10 — much closer than the original 13. The new loss is (9.052-10)² ≈ 0.90, down from 9. One backward pass and one small step cut the error to a tenth of what it was.

This is the point where a common mix-up happens. Backpropagation and gradient descent are often used as if they're the same thing — they are not. Backpropagation is the algorithm that computes the gradients (everything in the "Worked Example" section above: applying the chain rule backward through the network). Gradient descent is the separate, much simpler rule that uses those gradients to actually move the weights, shown just above. A full training step is always: forward pass → backpropagation (get gradients) → gradient descent (apply update). Backpropagation never touches a weight's value directly — it only ever produces the number that tells gradient descent which way and how far to move it.

Complete, Runnable Trace in Code

Every number above, reproduced in code. Run this yourself — the printed values should match the hand trace exactly.

x = 2
y = 10  # target

w1, b1 = 3, 1
w2, b2 = 2, -1

# ---- forward pass ----
z1 = w1 * x + b1        # 3*2 + 1 = 7
h  = max(0, z1)          # ReLU: 7
z2 = w2 * h + b2        # 2*7 - 1 = 13
o  = z2                  # linear output neuron

loss = (o - y) ** 2      # (13-10)**2 = 9
print(z1, h, z2, o, loss)
# 7 7 13 13 9

# ---- backward pass (backpropagation) ----
dL_do  = 2 * (o - y)          # 2*3 = 6
dL_dz2 = dL_do * 1              # linear activation: do/dz2 = 1
dL_dw2 = dL_dz2 * h              # 6*7 = 42
dL_db2 = dL_dz2 * 1              # 6
dL_dh  = dL_dz2 * w2            # 6*2 = 12
dh_dz1 = 1 if z1 > 0 else 0    # ReLU derivative: 1 since z1=7>0
dL_dz1 = dL_dh * dh_dz1          # 12*1 = 12
dL_dw1 = dL_dz1 * x              # 12*2 = 24
dL_db1 = dL_dz1 * 1              # 12

print(dL_dw1, dL_db1, dL_dw2, dL_db2)
# 24 12 42 6

# ---- gradient descent update ----
lr = 0.01
w1 -= lr * dL_dw1   # 3 - 0.24 = 2.76
b1 -= lr * dL_db1   # 1 - 0.12 = 0.88
w2 -= lr * dL_dw2   # 2 - 0.42 = 1.58
b2 -= lr * dL_db2   # -1 - 0.06 = -1.06

Because every value here — x, y, w1, b1, w2, b2 — starts as a plain integer, and 1/0 in the ReLU-derivative line are written as integers rather than 1.0/0.0, every gradient printed in that second line is a clean integer. This matters for a subtle reason: mixing an integer weight with a float derivative silently turns every downstream gradient into a float too, which is harmless mathematically but means your printed output won't visually match a hand trace unless you're careful about it — worth remembering the first time your own code's output "looks different" from your worked-out numbers despite being numerically identical.

The Dead ReLU: When a Gradient Is Zero

Look back at Step 5 of the hand trace: dh/dz1 was 1 only because z1 = 7 happened to be positive. What if it hadn't been?

Keep w1=3, b1=1, w2=2, b2=-1 but change the input to x = -5, target still y=10. Forward: z1 = 3×(-5)+1 = -14, so h = ReLU(-14) = 0. Then z2 = 2×0 - 1 = -1, o=-1, and loss = (-1-10)² = 121 — a much bigger error than before. Now trace the gradients backward: dL/do = 2×(-1-10) = -22, dL/dz2 = -22. So far normal. But now:

• dL/dw2 = dL/dz2 × h = -22 × 0 = 0
• dh/dz1 = 0 (since z1 = -14 < 0, ReLU is flat there)
• dL/dz1 = dL/dh × dh/dz1 = (-22×2) × 0 = 0
• dL/dw1 = 0 × x = 0, and dL/db1 = 0

Despite a loss of 121 — far worse than our original example — w1, b1, and even w2 all receive a gradient of exactly zero. Gradient descent will not move them at all this step, no matter how wrong the network is. This is called a dead ReLU: once a neuron's input is negative, its output is flat zero, its local slope is zero, and multiplying anything by zero anywhere in a chain-rule product zeroes out everything upstream of it, permanently blocking that path until (if ever) a different training example pushes z1 positive again. Only b2, which affects z2 directly rather than through h, still updates. This is a real, well-documented issue in networks that use ReLU — it's one reason engineers sometimes prefer variants like Leaky ReLU, which never has a perfectly flat, zero-slope region.

Practice: Test Yourself

Work these out by hand before checking the answers underneath. Use the same network structure throughout: z1=w1x+b1, h=ReLU(z1), z2=w2h+b2, o=z2, L=(o-y)².

1. With w1=3, b1=1, w2=2, b2=-1, run the full forward and backward pass for x=1, y=5. Find z1, h, o, L, then all four gradients.
2. Using the same weights, if x=-5 and instead y=-1 (so the network's wrong prediction happens to already equal the target's neighborhood), what would you expect about the size of dL/do compared to the dead-ReLU example above where y=10? You don't need to compute — reason about it from the formula dL/do = 2(o-y).
3. True or false, with justification: "Backpropagation updates the network's weights."
4. True or false, with justification: "A larger learning rate always makes training reach the correct weights faster."
5. In one or two sentences, explain why backpropagation computes gradients starting at the output layer and moving backward, rather than starting at the input layer and moving forward.

Answer Key

1. z1 = 3(1)+1 = 4, h = 4, z2 = 2(4)-1 = 7, o = 7, L=(7-5)²=4. Backward: dL/do=2(7-5)=4, dL/dz2=4, dL/dw2=4×4=16, dL/db2=4, dL/dh=4×2=8, dh/dz1=1 (since z1=4>0), dL/dz1=8, dL/dw1=8×1=8, dL/db1=8.
2. With x=-5, o is still -1 regardless of what y is (the forward pass doesn't involve y at all). If y=-1 too, then o-y=0, so dL/do=2(0)=0 — a much smaller gradient than the -22 we got when y=10, because the prediction is already correct and there is nothing to correct. The gradient's size directly reflects how wrong the prediction currently is.
3. False. Backpropagation only computes the gradients (how much each weight contributed to the error). It is gradient descent — a separate step — that actually changes the weight values using those gradients.
4. False. A learning rate that's too large can overshoot the correct weights entirely, causing the loss to jump around or even increase instead of settling down; there's a "sweet spot" range, not "bigger is always better."
5. Computing dL/dw1 requires already knowing dL/dz1 (chain rule), which itself requires dL/dh, which requires dL/dz2, and so on — each gradient is built from the one "after" it in the forward direction. There is no way to know how sensitive the final loss is to an early weight without first knowing how sensitive it is to everything that weight's effect passes through on its way to the output. So the only order that works is output first, input last — hence "backward."

Summary

• Backpropagation computes, for every weight and bias in a network, exactly how much a tiny change in it would change the final loss — its gradient — using the chain rule.
• The chain rule says the total sensitivity of the loss to a far-back weight is the product of the local sensitivities at every step along the path connecting them, computed one layer at a time, moving from the output back toward the input.
• Local sensitivities can be derived from basic algebra: expanding (o-y+ε)² gives dL/do=2(o-y); expanding (w+ε)h+b gives the sensitivity of a weighted sum to its weight as the other multiplied term.
• Backpropagation is not gradient descent. Backpropagation computes the gradients; gradient descent is the separate rule w_new = w_old - learning_rate × dL/dw that uses those gradients to actually move the weights.
• Going backward (rather than testing each weight separately) is what makes training large networks feasible: one forward pass plus one backward pass computes every gradient at once, by reusing shared intermediate sensitivities instead of recomputing them per weight.
• A ReLU neuron whose input is negative has a local slope of exactly zero, which zeroes out every gradient upstream of it in that chain-rule product — a "dead ReLU" that gradient descent cannot fix on that training step, no matter how large the loss is.