Building a Neural Network from Scratch: The Complete Implementation

Suppose you want to predict whether a student will clear a mock JEE test, using two numbers: hours studied this week (scaled to 0–1) and attendance percentage (also scaled to 0–1). You could try a single neuron — one weighted sum passed through an activation function — and for many students it would work fine. But now imagine the real pattern is stranger: students pass only if they have either very high study hours or very high attendance, but not some proportional mix of middling amounts of both — and a student with middling amounts of both actually fails, because neither habit is strong enough to compensate for the other. A single neuron draws exactly one straight decision boundary through the input space. It cannot bend around a pattern like that, no matter how you tune its two weights and one bias. This is not a hypothetical weakness — it is the exact failure that killed the first wave of neural network research in the 1960s, when Minsky and Papert proved a single-layer perceptron cannot represent the XOR function. The fix that revived the field is deceptively simple to state and remarkably intricate to compute: stack neurons into layers, and let the network learn how to combine them. This chapter builds that stack — every weight, every gradient, every update — by hand and then in code, so that by the end, "backpropagation" stops being a magic word and becomes a sequence of derivatives you could redo yourself with a pen.

Recap: what a single neuron computes

A single artificial neuron takes an input vector x = (x₁, x₂, ..., xₙ), multiplies each entry by a learned weight, adds a learned bias, and passes the result through a non-linear activation function:

z = w1*x1 + w2*x2 + ... + wn*xn + b
a = activation(z)

We will use the sigmoid activation throughout this chapter, defined as:

sigmoid(z) = 1 / (1 + e^(-z))

Sigmoid squashes any real number into the open interval (0, 1), which is why it is a natural choice when the output should behave like a probability — "how likely is this student to pass?" The problem, as the JEE example shows, is that one neuron only ever computes one weighted sum. Its decision boundary — the set of points where z = 0 — is always a straight line (or in higher dimensions, a flat hyperplane). To bend that boundary, we need more than one neuron working together, arranged in layers.

The architecture we will build

We are going to construct the smallest network that can actually bend a decision boundary: two input features, a hidden layer of two neurons, and one output neuron. This is usually written as a "2-2-1" network. Every input connects to every hidden neuron (this is called a fully connected or dense layer), and every hidden neuron connects to the output neuron. Each connection carries its own weight.

Notice the direction each pass travels. The forward pass (blue) starts at the inputs and pushes numbers rightward through the network until it produces a prediction. The backward pass (orange), which we will derive later in this chapter, starts at the output and pushes gradients — measurements of "how much did this weight contribute to the error" — leftward, layer by layer, back toward the inputs. This right-to-left flow of blame is the entire idea behind the word "back" in backpropagation, and the diagram's two directions are worth re-reading once we reach the backward-pass derivation.

The forward pass: computing a prediction step by step

Let's fix concrete numbers so every step is traceable. Take a student with 5 hours of study this week (scaled: x₁ = 0.5) and 80% attendance (scaled: x₂ = 0.8). Suppose this student actually passed, so the true label is y = 1. We initialize the network with these (arbitrary, small) starting weights and biases:

W1 = [[0.1, 0.2],   # weights into h1: from x1, from x2
      [0.3, 0.4]]   # weights into h2: from x1, from x2
b1 = [0.1, 0.1]

W2 = [0.5, 0.6]      # weights into y_hat: from h1, from h2
b2 = 0.2

Step 1 — hidden layer pre-activations. Each hidden neuron computes its own weighted sum of the two inputs:

z1_1 = 0.1(0.5) + 0.2(0.8) + 0.1 = 0.05 + 0.16 + 0.1 = 0.31
z1_2 = 0.3(0.5) + 0.4(0.8) + 0.1 = 0.15 + 0.32 + 0.1 = 0.57

Step 2 — hidden layer activations. Pass each through sigmoid:

a1_1 = sigmoid(0.31) = 1/(1+e^-0.31) = 1/1.7334 ≈ 0.5769
a1_2 = sigmoid(0.57) = 1/(1+e^-0.57) = 1/1.5655 ≈ 0.6388

Step 3 — output pre-activation. The output neuron treats the hidden layer's activations as its own inputs:

z2 = 0.5(0.5769) + 0.6(0.6388) + 0.2
   = 0.28845 + 0.38328 + 0.2 = 0.87173

Step 4 — output activation, the prediction.

y_hat = sigmoid(0.87173) = 1/(1+e^-0.87173) ≈ 1/1.4182 ≈ 0.7051

The network currently predicts a 70.5% chance the student passes. The true label is 1 (pass), so the network is in the right direction but under-confident — a small error, not a wild one, which is exactly what we'd expect from an untrained network that happened to start with sensible-looking weights. Every subsequent step of this chapter exists to answer one question: how should each of the six weights and three biases change to make y_hat move closer to 1?

Measuring wrongness: the loss function

Before we can improve the prediction, we need a single number that tells us how wrong it is. We'll use the squared error loss for this one example:

L = (1/2) * (y_hat - y)^2

Squaring serves two purposes. First, it makes the loss non-negative regardless of whether the prediction overshoots or undershoots the true label — (y_hat − y)² treats a 0.3 overshoot and a 0.3 undershoot identically, which is the correct behaviour: both are equally wrong. Second, squaring makes larger errors cost disproportionately more than smaller ones, which pushes training to eliminate big mistakes quickly. The 1/2 factor is a bookkeeping convenience: when we differentiate the squared term, the power rule brings down a factor of 2, which the 1/2 exactly cancels, leaving a clean dL/dy_hat = (y_hat − y) instead of 2(y_hat − y). Nothing about the network's behaviour depends on this constant — it is purely there to make the derivative tidy.

Plugging in our numbers:

L = 0.5 * (0.7051 - 1)^2 = 0.5 * (-0.2949)^2 = 0.5 * 0.08696 ≈ 0.0435

The chain rule intuition: how blame flows backward

Here is the central difficulty. The loss L depends on y_hat, which depends on z2, which depends on a1 and W2, which depend on z1, which depends on W1 and the original inputs. If we want to know how a tiny nudge to, say, W1's very first entry affects the final loss, we have to trace that nudge through every intermediate computation it passes through. This is exactly what the calculus chain rule is built for: if L is a function of y_hat, and y_hat is a function of z2, then

dL/dz2 = (dL/dy_hat) * (dy_hat/dz2)

Think of it as a relay race of local slopes. Each computational step in the network only "knows" its own local derivative — how sensitive its output is to its own input. The chain rule says that to find the sensitivity of the loss to something several steps upstream, you simply multiply the local slopes along the path connecting them. This is also why the algorithm is called backpropagation rather than just "propagation": we compute these local derivatives starting from the loss (the end of the chain) and multiply our way backward toward the earliest weights, because each new local derivative we need is naturally expressed in terms of the one we just computed one step closer to the output. Going forward through the chain would force us to recompute the same downstream derivatives over and over for every single weight; going backward, each weight reuses work already done for the layer after it.

Deriving the sigmoid derivative

Since every layer in our network uses sigmoid, we need its derivative once, carefully, and then we can reuse it everywhere. Starting from s(z) = 1/(1+e^-z) = (1+e^-z)^-1, apply the chain rule (differentiating the outer power, then the inner exponential):

ds/dz = -1 * (1+e^-z)^-2 * (-e^-z)
      = e^-z / (1+e^-z)^2

Now rewrite the numerator as (1 + e^-z) − 1, a small algebraic trick that lets us split the fraction:

ds/dz = [(1+e^-z) - 1] / (1+e^-z)^2
      = 1/(1+e^-z) - 1/(1+e^-z)^2
      = s(z) - s(z)^2
      = s(z) * (1 - s(z))

This is the single most useful fact in this chapter: the derivative of sigmoid, evaluated at z, equals sigmoid(z) times (1 − sigmoid(z)). Because we already computed every activation value in the forward pass, we never need to touch z or e^-z again during backpropagation — we just plug the already-known activation a into a(1−a).

Backpropagation: computing every gradient by hand

We now compute dL/dw for all six weights and dL/db for all three biases, working from the output backward to the first layer, reusing each result in the next.

Output layer. By the chain rule, the sensitivity of the loss to the output's pre-activation z2 is:

delta2 = dL/dz2 = (dL/dy_hat) * (dy_hat/dz2)
       = (y_hat - y) * y_hat * (1 - y_hat)
       = (0.7051 - 1) * 0.7051 * 0.2949
       = (-0.2949) * (0.2079)
       ≈ -0.06131

We call this quantity delta2 because it is the "error signal" that will get distributed backward from the output neuron. Once we have it, the gradient with respect to any weight or bias feeding into z2 is simply delta2 times whatever fed into that particular connection (this follows because z2 = w2_1*a1_1 + w2_2*a1_2 + b2, so ∂z2/∂w2_1 = a1_1, and so on):

dL/dw2_1 = delta2 * a1_1 = -0.06131 * 0.5769 ≈ -0.03537
dL/dw2_2 = delta2 * a1_2 = -0.06131 * 0.6388 ≈ -0.03917
dL/db2   = delta2                               ≈ -0.06131

Hidden layer. Now we push the error signal one layer further back. Each hidden neuron's activation a1_i affected the loss only through its contribution to z2, weighted by w2_i, so:

delta1_i = (delta2 * w2_i) * a1_i * (1 - a1_i)

delta1_1 = (-0.06131 * 0.5) * 0.5769 * 0.4231 ≈ -0.007483
delta1_2 = (-0.06131 * 0.6) * 0.6388 * 0.3612 ≈ -0.008486

Notice the structure repeats exactly: each hidden neuron's error signal is (the error signal flowing in from downstream, weighted by the connecting weight) times (its own local sigmoid slope). This is the chain rule applied one more layer back, and it is the pattern that lets backpropagation scale to networks with dozens of layers — the same two-step recipe (weight the incoming error, multiply by the local slope) repeats at every layer, regardless of how many there are.

With delta1 in hand, the gradients for the first layer's weights and biases follow the same rule as before — each gradient is the neuron's error signal times whatever value fed that particular connection (here, the raw inputs x):

dL/dW1_11 = delta1_1 * x1 = -0.007483 * 0.5 ≈ -0.003742
dL/dW1_12 = delta1_1 * x2 = -0.007483 * 0.8 ≈ -0.005986
dL/dW1_21 = delta1_2 * x1 = -0.008486 * 0.5 ≈ -0.004243
dL/dW1_22 = delta1_2 * x2 = -0.008486 * 0.8 ≈ -0.006789

dL/db1_1 = delta1_1 ≈ -0.007483
dL/db1_2 = delta1_2 ≈ -0.008486

Every one of the nine gradients the network needs has now been computed, using nothing beyond arithmetic and the chain rule applied twice.

Gradient descent: the update rule

A gradient tells us the direction of steepest increase of the loss. To reduce the loss, we move each parameter a small step in the opposite direction, scaled by a learning rate η that controls how big the step is:

w_new = w_old - η * (dL/dw)

Take η = 0.5 for this example (an aggressively large rate, chosen so the update is easy to check by hand — real networks typically use something far smaller, like 0.01). Since every gradient we computed above is negative, and we subtract a negative number, every weight increases slightly — which is exactly right, because y_hat needs to move up toward 1:

w2_1: 0.5 - 0.5(-0.03537) = 0.5177
w2_2: 0.6 - 0.5(-0.03917) = 0.6196
b2:   0.2 - 0.5(-0.06131) = 0.2307

W1_11: 0.1 - 0.5(-0.003742) = 0.1019
W1_12: 0.2 - 0.5(-0.005986) = 0.2030
W1_21: 0.3 - 0.5(-0.004243) = 0.3021
W1_22: 0.4 - 0.5(-0.006789) = 0.4034

b1_1: 0.1 - 0.5(-0.007483) = 0.1037
b1_2: 0.1 - 0.5(-0.008486) = 0.1042

If you re-ran the forward pass with these updated numbers, y_hat would come out slightly higher than 0.7051 — closer to the target of 1 — and the loss would be slightly lower than 0.0435. One student, one update, one small improvement. A full training run repeats this forward-loss-backward-update cycle thousands of times across many students, and it is this repetition, not any single step, that eventually produces a network whose weights encode a genuinely non-linear boundary between "pass" and "fail."

Implementing it in Python, from scratch

The code below implements exactly the computation above — no machine learning library, just math.exp. Running it reproduces every number derived by hand in the previous sections.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def sigmoid_derivative(a):
    # a must already be a sigmoid OUTPUT (not the raw z)
    return a * (1 - a)

# ---- network parameters: 2 inputs, 2 hidden neurons, 1 output ----
W1 = [[0.1, 0.2],   # weights into h1: [from x1, from x2]
      [0.3, 0.4]]   # weights into h2: [from x1, from x2]
b1 = [0.1, 0.1]

W2 = [0.5, 0.6]      # weights into y_hat: [from h1, from h2]
b2 = 0.2

x = [0.5, 0.8]        # hours studied (scaled), attendance (scaled)
y = 1                 # true label: 1 = passed
learning_rate = 0.5

# ---------------- forward pass ----------------
z1 = [W1[i][0]*x[0] + W1[i][1]*x[1] + b1[i] for i in range(2)]
a1 = [sigmoid(z) for z in z1]

z2 = W2[0]*a1[0] + W2[1]*a1[1] + b2
y_hat = sigmoid(z2)

loss = 0.5 * (y_hat - y) ** 2
print(f"a1 = {a1}")
print(f"y_hat = {y_hat:.4f}, loss = {loss:.4f}")

# ---------------- backward pass ----------------
delta2 = (y_hat - y) * sigmoid_derivative(y_hat)

dW2 = [delta2 * a1[0], delta2 * a1[1]]
db2 = delta2

delta1 = [delta2 * W2[i] * sigmoid_derivative(a1[i]) for i in range(2)]
dW1 = [[delta1[i] * x[j] for j in range(2)] for i in range(2)]
db1 = [delta1[i] for i in range(2)]

# ---------------- parameter update ----------------
W2 = [W2[i] - learning_rate * dW2[i] for i in range(2)]
b2 = b2 - learning_rate * db2
W1 = [[W1[i][j] - learning_rate * dW1[i][j] for j in range(2)] for i in range(2)]
b1 = [b1[i] - learning_rate * db1[i] for i in range(2)]

print(f"updated W2 = {[round(v,4) for v in W2]}, b2 = {b2:.4f}")
print(f"updated W1 = {[[round(v,4) for v in row] for row in W1]}")
print(f"updated b1 = {[round(v,4) for v in b1]}")

Tracing the printed output against the hand-worked values above: a1 prints approximately [0.5769, 0.6388], y_hat prints 0.7051 with loss at 0.0435, and the updated parameters print W2 = [0.5177, 0.6196], b2 = 0.2307, W1 = [[0.1019, 0.2030], [0.3021, 0.4034]], and b1 = [0.1037, 0.1042] — matching every number derived by hand, digit for digit. To train on a real dataset rather than one example, you would wrap the forward-backward-update block in a loop over every (x, y) pair in the training data, repeated for many epochs (one epoch = one full pass through all the training examples), which is precisely what libraries like PyTorch or TensorFlow automate — they do not use a different algorithm, they compute this same chain-rule bookkeeping automatically (a technique called autodiff) so you don't have to derive delta1 and delta2 by hand for every architecture you design.

Why can't we just initialize all weights to zero?

A very natural instinct, especially if you're used to setting counters or accumulators to zero before a loop, is to initialize every weight in the network to 0 rather than to arbitrary small numbers like the 0.1–0.6 we used above. This instinct is wrong, and understanding exactly why is more instructive than the fact itself. Suppose W1 = [[0,0],[0,0]] and b1 = [0,0]. Then z1_1 and z1_2 are both 0, so a1_1 = a1_2 = sigmoid(0) = 0.5 — the two hidden neurons produce identical outputs. Now look at the backward pass: delta1_1 and delta1_2 both depend on a1_i(1−a1_i), which is identical for both neurons, and both are scaled by the same downstream delta2. The gradients dL/dW1 for the row feeding h1 and the row feeding h2 come out exactly equal. When we subtract equal gradients from equal weights, h1 and h2 stay identical after the update — and by the same argument, they stay identical after every future update, forever. Two neurons that are permanently forced to compute the same function are, for all practical purposes, one neuron wearing two name tags — the network never gains the representational power that having two neurons was supposed to buy it. This failure is called the symmetry problem, and it's why real implementations initialize weights to small random values (breaking the symmetry so different neurons drift toward learning different features) while biases can safely start at zero, since the weights alone already break the symmetry between neurons in the same layer.

Scaling up: epochs, batches, and what actually changes

Everything above was one weight update from one training example. A real training run differs only in scale, not in kind. Instead of one (x, y) pair, you have a dataset of thousands. Instead of one epoch, you loop over the full dataset dozens or hundreds of times. And instead of updating weights after every single example (called stochastic gradient descent, or SGD), most practical systems average the gradients over a small batch of examples — say 32 or 64 at a time — before applying one update, which produces a smoother, less noisy descent toward low loss. Nothing about the chain-rule mathematics changes: batching just means computing delta2 and delta1 for each example in the batch separately, then averaging the resulting dW1, dW2, db1, db2 before the single update step. The architecture can also grow — more hidden neurons per layer, more hidden layers stacked before the output — and the backward pass simply repeats the same two-step recipe (weight the incoming delta by the connecting weight, multiply by the local activation slope) once per additional layer, which is exactly why this algorithm scales from the 2-2-1 toy network in this chapter to networks with hundreds of layers used in production systems.

CBSE and competitive exam connections

The differentiation techniques used throughout this chapter — the chain rule and the quotient-rule-style derivation of the sigmoid function — sit squarely inside the Class 11–12 Applications of Derivatives and Continuity & Differentiability syllabus, and they appear directly in IIT-JEE (Main and Advanced) and BITSAT calculus sections, usually phrased as "differentiate f(g(x))" problems rather than in a neural-network context. If you can reproduce the sigmoid derivative step by step the way we did here, you have effectively practiced a JEE-level composite-function differentiation problem while also understanding where it's used. The gradient descent update rule (w_new = w_old − η · dL/dw) is a first, concrete instance of an idea you'll meet formally as multivariable optimization if you pursue engineering or the KVPY/Olympiad track — finding a minimum by moving opposite to a derivative generalizes directly to partial derivatives once a function depends on many variables at once, which is exactly what the nine parameters of our tiny network already are. Treat this chapter as an early, hands-on encounter with an idea your calculus courses will later name formally.

Check your understanding

• Using the updated weights from this chapter (W2 = [0.5177, 0.6196], b2 = 0.2307, W1 = [[0.1019, 0.2030],[0.3021, 0.4034]], b1 = [0.1037, 0.1042]), run the forward pass again on the same student (x = [0.5, 0.8]). Is the new y_hat higher than 0.7051? Is the new loss lower than 0.0435? (You should find both are true, by a small amount.)
• A second student has x = [0.2, 0.3] and true label y = 0. Using the original weights from the start of this chapter, compute z1, a1, z2, and y_hat for this student. Is the network's initial prediction closer to correct for this student or the first one?
• Explain in your own words why delta1_i is multiplied by w2_i (the weight connecting hidden neuron i to the output) rather than by w2 of the other hidden neuron. What would go wrong if you swapped them?
• If a network's hidden layer had 3 neurons instead of 2, all initialized to identical non-zero weight rows, would the symmetry problem still occur? Justify your answer using the argument from the "why can't we just initialize all weights to zero" section.
• Suppose you used a learning rate of η = 50 instead of 0.5 in this chapter's example. Predict, without recomputing every decimal, what is likely to go wrong with the updated weights, and why.

Summary

A neural network's forward pass is nothing more than repeated weighted sums followed by an activation function, layer after layer, until a final number emerges as the prediction. Backpropagation is not a separate, mysterious algorithm — it is the calculus chain rule, applied systematically from the output backward to the first layer, so that each layer's error signal (delta) can be computed by reusing the layer after it, rather than starting from scratch. The squared-error loss gives us a single number to minimize; its derivative with respect to the prediction is what starts the backward chain. The sigmoid function's clean derivative, s(1−s), is what makes each layer's local computation tractable by hand. Gradient descent then converts every computed gradient into a small, direction-correct nudge to a weight. None of the six weights or three biases in our 2-2-1 network moved by more than about 0.02 in a single update — real training requires thousands of such small nudges across thousands of examples before a network's decision boundary bends into genuinely useful shape. What changed today is not that the network learned instantly, but that every one of those nudges is now something you can compute, trace, and verify by hand, rather than a black box you trust blindly.