Backpropagation: The Algorithm That Powers Deep Learning

Suppose you build a tiny neural network to predict whether a student clears a mock-test cutoff, using just two numbers: attendance fraction and average practice-test score. The network has one hidden layer of two neurons and one output neuron — three computing neurons in total, wired together by 9 trainable numbers (6 weights and 3 biases). You run it, and it predicts 0.76 when the true answer should be close to 1. The prediction is wrong. Now what?

You need to nudge all 9 numbers so the prediction gets closer to correct. The question that this entire chapter answers is: nudge each one by how much, and in which direction? Some of those 9 numbers are directly connected to the wrong output; others are two layers removed from it, connected only through neurons that are themselves connected through other neurons. A weight buried in the first layer has no direct line of sight to the error — its effect on the loss is filtered through everything downstream of it. Backpropagation is the algorithm that solves exactly this problem: it computes the exact contribution of every single weight to the final error, no matter how deep it sits, using one clean mathematical idea applied over and over — the chain rule of calculus.

Why you can't just try every weight one at a time

There is a naive way to find out how much a weight matters: nudge it by a tiny amount ε, rerun the whole network, see how much the loss changed, then divide. This is called numerical differentiation, and it works — but it costs one full forward pass per weight. Real networks have millions or billions of weights. A modern language model has on the order of 10¹¹ weights; even at a generous one microsecond per forward pass, checking every weight this way would take longer than you have. Backpropagation computes the exact same derivatives for all weights in roughly the cost of two passes through the network — one forward, one backward — regardless of how many weights there are. That efficiency, not just correctness, is why it is the algorithm that made deep learning computationally possible at all.

The worked network we'll use throughout

Here is the exact network we will hand-compute, so every formula in this chapter has real numbers attached to it.

• Inputs: x₁ = 0.50 (attendance fraction), x₂ = 0.90 (practice-score fraction)
• Target: y = 1 (this student should be predicted to clear the cutoff)
• Hidden layer: two sigmoid neurons, h1 and h2
• Output layer: one sigmoid neuron, o
• Weights into h1: w1 = 0.15 (from x1), w2 = 0.20 (from x2), bias b1 = 0.35
• Weights into h2: w3 = 0.25 (from x1), w4 = 0.30 (from x2), bias b2 = 0.35
• Weights into o: w5 = 0.40 (from h1), w6 = 0.45 (from h2), bias b3 = 0.60

That is 6 weights plus 3 biases — the 9 numbers we opened with. Every neuron computes a weighted sum z, then squashes it with the sigmoid function σ(z) = 1 / (1 + e⁻ᶻ), which maps any real number to a value strictly between 0 and 1 — convenient for a "probability the student clears the cutoff" style output. We measure error with the squared-error loss L = ½(y − a_o)², where a_o is the network's output.

Step 1: the forward pass

Before you can assign blame, you have to see what the network actually produced. This is the forward pass — plain arithmetic, no calculus yet.

z_h1 = w1·x1 + w2·x2 + b1 = 0.15(0.50) + 0.20(0.90) + 0.35 = 0.6050
a_h1 = σ(z_h1) = σ(0.6050) = 0.6468

z_h2 = w3·x1 + w4·x2 + b2 = 0.25(0.50) + 0.30(0.90) + 0.35 = 0.7450
a_h2 = σ(z_h2) = σ(0.7450) = 0.6781

z_o  = w5·a_h1 + w6·a_h2 + b3 = 0.40(0.6468) + 0.45(0.6781) + 0.60 = 1.1639
a_o  = σ(z_o) = σ(1.1639) = 0.7620

L = ½(y − a_o)² = ½(1 − 0.7620)² = 0.0283

The network predicted 0.7620 against a target of 1, giving a loss of 0.0283. Not catastrophic, but there is clear room to improve — and improving means adjusting all 9 weights in the direction that reduces L. That direction is given by the negative gradient, so we need ∂L/∂w for every one of the 9 weights.

Step 2: the one derivative you must derive first — the sigmoid's own slope

Every gradient calculation below needs to know how steeply the sigmoid curve is rising at the point we evaluated it. Rather than quote this, derive it, because the result has a strange and important property.

Start from σ(z) = (1 + e⁻ᶻ)⁻¹ and differentiate with the chain rule:

σ'(z) = −1·(1 + e⁻ᶻ)⁻² · (−e⁻ᶻ) = e⁻ᶻ / (1 + e⁻ᶻ)²

Now rewrite e⁻ᶻ as (1 + e⁻ᶻ) − 1, and split the fraction:

σ'(z) = [(1 + e⁻ᶻ) − 1] / (1 + e⁻ᶻ)²
      = 1/(1 + e⁻ᶻ) − 1/(1 + e⁻ᶻ)²
      = σ(z) − σ(z)²
      = σ(z)·(1 − σ(z))

This is the fact you'll use dozens of times: the derivative of the sigmoid, at any point, equals the sigmoid's own output at that point times one minus that output. If a neuron's activation is a, its local slope is simply a(1 − a) — no need to remember z at all once you have a. Notice also that a(1 − a) is a downward parabola in a, maximized when a = 0.5, where it equals 0.5 × 0.5 = 0.25. The sigmoid's slope can never exceed 0.25, anywhere. Hold onto that number — it resurfaces later as the root cause of a real problem in deep networks.

Step 3: the backward pass — assigning blame with the chain rule

We now compute ∂L/∂w for each of the 9 weights, starting from the output and working backward — which is where the algorithm gets its name. The central tool is the chain rule: if L depends on a, and a depends on z, and z depends on w, then

∂L/∂w = (∂L/∂a) · (∂a/∂z) · (∂z/∂w)

Each factor is something you already know how to compute locally, at each neuron, without needing to see the whole network at once. That locality is what makes the algorithm efficient.

Output layer first. Define δ_out = ∂L/∂z_o — how sensitive the loss is to the output neuron's pre-activation sum. By the chain rule, δ_out = (∂L/∂a_o)·(∂a_o/∂z_o). From L = ½(y − a_o)², ∂L/∂a_o = −(y − a_o) = a_o − y. Combined with the sigmoid derivative from Step 2:

δ_out = (a_o − y) · a_o(1 − a_o)
      = (0.7620 − 1) · 0.7620 · (1 − 0.7620)
      = (−0.2380) · (0.1814)
      = −0.0432

Once you have δ_out, the gradient for any weight feeding into o is just δ_out times whatever activation traveled along that specific connection — because z_o = w5·a_h1 + w6·a_h2 + b3, so ∂z_o/∂w5 = a_h1, ∂z_o/∂w6 = a_h2, and ∂z_o/∂b3 = 1:

∂L/∂w5 = δ_out · a_h1 = −0.0432 × 0.6468 = −0.02791
∂L/∂w6 = δ_out · a_h2 = −0.0432 × 0.6781 = −0.02926
∂L/∂b3 = δ_out · 1     = −0.04315

Now the hidden layer — the step that makes this "back-propagation". To get ∂L/∂w1, you need to know how much h1's pre-activation sum z_h1 affects the loss. But z_h1 doesn't touch the loss directly — it only affects L by first changing a_h1, which changes z_o, which changes a_o, which changes L. That's a longer chain, but it's still just the chain rule, now with more links:

δ_h1 = ∂L/∂z_h1 = (∂L/∂z_o) · (∂z_o/∂a_h1) · (∂a_h1/∂z_h1)
     = δ_out · w5 · a_h1(1 − a_h1)

Read this literally: the error signal at the output (δ_out) travels backward through the connection weight it came through (w5, because that's how much a_h1 influenced z_o), then gets scaled by h1's own local sigmoid slope. Every δ in every layer is built this way — take the δ from the layer ahead, multiply by the weight connecting the two neurons, multiply by the local slope. Plugging in numbers:

δ_h1 = δ_out · w5 · a_h1(1 − a_h1) = (−0.0432)(0.40)(0.6468)(0.3532) = −0.0039
δ_h2 = δ_out · w6 · a_h2(1 − a_h2) = (−0.0432)(0.45)(0.6781)(0.3219) = −0.0042

And now, exactly as before, each weight's gradient is its δ times the activation that fed it (or times 1, for a bias):

∂L/∂w1 = δ_h1 · x1 = (−0.0039)(0.50) = −0.00197
∂L/∂w2 = δ_h1 · x2 = (−0.0039)(0.90) = −0.00355
∂L/∂b1 = δ_h1        = −0.00394

∂L/∂w3 = δ_h2 · x1 = (−0.0042)(0.50) = −0.00212
∂L/∂w4 = δ_h2 · x2 = (−0.0042)(0.90) = −0.00381
∂L/∂b2 = δ_h2        = −0.00424

All 9 gradients are now known, computed from just two passes over the network — one forward (Step 1) and one backward (this step) — rather than 9 separate perturbation experiments. This is the entire algorithm. Everything else — convolutional layers, transformers, attention — reuses this exact same chain-rule bookkeeping; only the shape of the connections changes.

Step 4: updating the weights and confirming the loss actually drops

Gradient descent moves every weight a small step against its gradient: w_new = w_old − η·(∂L/∂w), where η is the learning rate. Using η = 0.5:

w1 = 0.15 − 0.5(−0.00197) = 0.1510      w3 = 0.25 − 0.5(−0.00212) = 0.2511
w2 = 0.20 − 0.5(−0.00355) = 0.2018      w4 = 0.30 − 0.5(−0.00381) = 0.3019
b1 = 0.35 − 0.5(−0.00394) = 0.3520      b2 = 0.35 − 0.5(−0.00424) = 0.3521

w5 = 0.40 − 0.5(−0.02791) = 0.4140
w6 = 0.45 − 0.5(−0.02926) = 0.4646
b3 = 0.60 − 0.5(−0.04315) = 0.6216

Re-running the forward pass with these updated weights: a_h1 = 0.6477, a_h2 = 0.6790, a_o = 0.7695, giving L = 0.0266. The loss fell from 0.0283 to 0.0266 after a single update — proof, in numbers you can check by hand, that the gradients we computed genuinely point in a direction that reduces error. Repeat this forward-backward-update cycle thousands of times, typically over batches of many examples at once, and this is what "training a neural network" means at the arithmetic level — there is no additional magic hiding beneath it.

The same computation, verified in code

Here is the identical example as runnable Python. Trace it against the hand-worked numbers above — every printed value should match exactly.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def sigmoid_derivative(a):
    return a * (1 - a)

x1, x2 = 0.50, 0.90
y = 1.0

w1, w2, b1 = 0.15, 0.20, 0.35
w3, w4, b2 = 0.25, 0.30, 0.35
w5, w6, b3 = 0.40, 0.45, 0.60

# forward pass
z_h1 = w1*x1 + w2*x2 + b1;  a_h1 = sigmoid(z_h1)
z_h2 = w3*x1 + w4*x2 + b2;  a_h2 = sigmoid(z_h2)
z_o  = w5*a_h1 + w6*a_h2 + b3;  a_o = sigmoid(z_o)
L = 0.5 * (y - a_o) ** 2
print(f"a_h1={a_h1:.4f} a_h2={a_h2:.4f} a_o={a_o:.4f} L={L:.4f}")

# backward pass
delta_out = (a_o - y) * sigmoid_derivative(a_o)
delta_h1  = delta_out * w5 * sigmoid_derivative(a_h1)
delta_h2  = delta_out * w6 * sigmoid_derivative(a_h2)
print(f"delta_out={delta_out:.4f} delta_h1={delta_h1:.4f} delta_h2={delta_h2:.4f}")

grad = {
    "w5": delta_out * a_h1, "w6": delta_out * a_h2, "b3": delta_out,
    "w1": delta_h1 * x1, "w2": delta_h1 * x2, "b1": delta_h1,
    "w3": delta_h2 * x1, "w4": delta_h2 * x2, "b2": delta_h2,
}
for k, v in grad.items():
    print(f"d{k}={v:.5f}")

Running this prints:

a_h1=0.6468 a_h2=0.6781 a_o=0.7620 L=0.0283
delta_out=-0.0432 delta_h1=-0.0039 delta_h2=-0.0042
dw5=-0.02791
dw6=-0.02926
db3=-0.04315
dw1=-0.00197
dw2=-0.00355
db1=-0.00394
dw3=-0.00212
dw4=-0.00381
db2=-0.00424

Every line matches the hand computation above digit for digit — the point being that backpropagation is not a metaphor or an approximation of what code does; the code above is the algorithm, and the algebra above is a proof that the code is correct.

Seeing it: forward and backward through the same network

The diagram below shows the identical 5-neuron network (2 inputs, 2 hidden, 1 output) twice: activations flowing left-to-right on top (the forward pass), and error signal δ flowing right-to-left on the bottom (the backward pass) along the exact same connections.

Three misconceptions worth killing now

Misconception 1: "Backpropagation and gradient descent are the same thing." They are not. Backpropagation is a method for computing the gradient — the vector of all ∂L/∂w values — efficiently, using the chain rule. Gradient descent is a separate method for using that gradient to update the weights, by stepping against it. You could compute the gradient with backpropagation and then update weights with a more sophisticated optimizer than plain gradient descent — Adam and RMSprop are common choices in real training pipelines — and you would still be using backpropagation. The two ideas are frequently taught together because they're always used together, but one is a differentiation technique and the other is an optimization technique.

Misconception 2: "δ is the error, i.e., the same thing as the loss." They are related but distinct quantities with different jobs. L is a single scalar number describing how wrong the entire network's output is on one example. δ (delta) is defined per neuron, as ∂L/∂z — how sensitive the loss is to that specific neuron's pre-activation sum. A network with even a handful of neurons — like the three computing neurons (2 hidden + 1 output) in our worked example — has multiple δ values (one per neuron: δ_out, δ_h1, δ_h2) but exactly one L. δ is a local, per-neuron blame signal computed by working backward from L; it is not L itself, and different neurons in the same network almost always carry different δ values, as our own numbers show (δ_out = −0.0432, δ_h1 = −0.0039, δ_h2 = −0.0042 — three different numbers from one loss).

Misconception 3: "Backpropagation trains one weight, then the next, then the next." It does not proceed weight-by-weight in sequence with separate forward passes for each. A single forward pass computes every activation; a single backward pass, sweeping layer by layer from the output toward the input, computes every δ and hence every gradient, simultaneously, by reusing the δ from the layer just processed. That reuse — δ_h1 and δ_h2 both being built directly from the single already-computed δ_out, rather than each being recomputed from scratch — is precisely what makes the algorithm cost one backward pass instead of nine separate ones. Sequential, weight-at-a-time computation is exactly the slow numerical-differentiation approach this algorithm was invented to avoid.

Why very deep networks can struggle: the vanishing gradient

Look again at how δ_h1 was built: δ_out · w5 · a_h1(1 − a_h1). The factor a_h1(1 − a_h1) is the sigmoid's local slope, and we proved earlier it can never exceed 0.25. Every additional hidden layer you stack introduces one more multiplication by a factor that is at most 0.25. Chain ten such layers together and the accumulated shrinkage factor is at most 0.25¹⁰ ≈ 0.00000095 — under one part in a million. In a very deep sigmoid network, the gradient reaching the earliest layers can become so small that gradient descent effectively stalls there; those weights barely move, no matter how many training steps you run. This is the vanishing gradient problem, and it's a direct, provable consequence of the sigmoid derivative's 0.25 ceiling combined with the chain rule's habit of multiplying local slopes together across layers. It's also the concrete reason modern deep networks mostly use ReLU-family activations instead of sigmoid in their hidden layers — ReLU's derivative is either 0 or exactly 1, so it doesn't shrink gradients on every layer the way sigmoid does.

Where this sits in your exams

CBSE's Artificial Intelligence syllabus (Code 417) places neural networks and the idea of a network "learning from error" explicitly on the curriculum, and backpropagation is the mechanism examiners expect you to be able to describe conceptually, layer by layer. For JEE Main/Advanced and BITSAT, the actual mathematical skill exercised here — applying the chain rule through several nested functions while keeping careful track of which derivative belongs to which stage — is precisely the kind of multi-step differentiation their calculus sections test, just dressed in different variable names. If you enjoy chain-rule-heavy problems, this worked example is good deliberate practice for that skill in general, independent of the AI context. At the GATE-foundation level, "multi-layer perceptron" and "feed-forward neural network" appear directly under the Machine Learning topics in the GATE Computer Science syllabus, and backpropagation is the standard algorithm assumed when those topics are examined.

Active recall

1. In our worked example, if you doubled w5 from 0.40 to 0.80 while holding everything else at its original value and recomputed the forward pass, would δ_h1 increase or decrease in magnitude? Work from the formula δ_h1 = δ_out · w5 · a_h1(1 − a_h1) rather than re-deriving everything from scratch. Self-check: w5 appears as a direct multiplying factor in δ_h1's formula, and a_o would also shift slightly since z_o depends on w5 too — but the dominant, easy-to-see effect is that a larger w5 directly scales up |δ_h1|, assuming δ_out doesn't flip sign or shrink faster than w5 grows.
2. Explain in one sentence why ∂L/∂w1 needed δ_h1 as an intermediate quantity, while ∂L/∂w5 could be computed directly from δ_out alone. Self-check: w1 is two layers removed from L, so the chain rule has to pass through an extra node (h1) that w5 doesn't have to pass through — w5 connects directly to the output neuron whose δ we already had.
3. If a network used a hidden activation whose derivative could reach a maximum of 1.0 instead of sigmoid's 0.25, would the vanishing-gradient effect over 10 layers be better or worse than the 0.25¹⁰ figure computed above? Give the analogous product. Self-check: 1.0¹⁰ = 1.0 — no shrinkage at all, which is a large part of why ReLU-style activations (max derivative 1) resist vanishing gradients far better than sigmoid does.
4. Without recomputing every number, state which of the 9 gradients in our example would change if only the target y were changed from 1 to 0, and explain why the rest would not. Self-check: every gradient depends on δ_out, and δ_out depends on y through (a_o − y) — so changing y changes δ_out, and since δ_h1, δ_h2 are both built from δ_out, changing y ripples through and changes literally all 9 gradients, not just the output-layer ones.

Summary

Backpropagation computes the gradient of the loss with respect to every weight in a network by applying the chain rule backward from the output, one layer at a time, reusing each layer's error signal δ to build the layer before it — turning what would be millions of expensive separate calculations into one forward pass plus one backward pass. The core building block is δ_layer = δ_next · weight · local_slope, and for sigmoid neurons that local slope is a(1 − a), a quantity that maxes out at 0.25 and is the direct cause of vanishing gradients in deep sigmoid networks. Everything from a two-hidden-neuron toy network to a modern deep transformer is trained by the same underlying arithmetic; only the wiring diagram changes.

Think About It

Think about this: How would you explain backpropagation: the algorithm that powers deep learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.