Building a Neural Network from Scratch in Python

In most Indian apartment buildings and hostels, a staircase light is wired to two switches — one at the bottom of the stairs, one at the top. Flip either switch and the light changes state. If both switches are down, the light is off. Flip the top one, it turns on. Flip the bottom one too, it turns off again. Flip the top one back, it's on. The light is on exactly when the two switches disagree, and off when they agree. Electricians call this a two-way switch circuit. In logic, it is the XOR (exclusive-OR) function, and it is about to break the simplest possible neural network — a single artificial neuron — before we've even started. Understanding exactly how and why it breaks is what will force us to invent the multi-layer network this chapter builds, line by line, in working Python.

A single neuron is a weighted vote

Strip a neuron down to its bare mechanics and it does one thing: it takes some numbers in, multiplies each by a weight, adds them up along with a bias term, and passes the result through a small nonlinear function. For two inputs x1 and x2:

z = w1*x1 + w2*x2 + b

w1 and w2 are weights the network will eventually learn; b is a bias that shifts the decision. For now, imagine the neuron fires (outputs 1) when z >= 0 and stays silent (outputs 0) otherwise — this is the original 1943 McCulloch-Pitts threshold neuron, the ancestor of every network you'll ever train. With the right weights, one neuron can compute AND: set w1 = w2 = 1, b = -1.5. Then z = x1 + x2 - 1.5, which is only >= 0 when both inputs are 1. Set b = -0.5 instead and the same neuron computes OR, firing whenever at least one input is 1.

Now look at what that condition w1*x1 + w2*x2 + b = 0 actually is. Rearranged, it's x2 = -(w1/w2)*x1 - b/w2 — the equation of a straight line in the x1-x2 plane, in exactly the slope-intercept form you already use for linear equations in two variables. A single neuron's decision rule is: which side of this line is the point on? Everything a lone neuron can ever learn is a straight-line split of the input plane into two regions.

Why XOR defeats a single neuron

Plot the four possible switch inputs as points on a plane, with x1 and x2 as the two switch positions (0 = down, 1 = up) and the label as whether the light is on:

• (0, 0) → light off → class 0
• (0, 1) → light on → class 1
• (1, 0) → light on → class 1
• (1, 1) → light off → class 0

The two "on" points sit on one diagonal of the unit square; the two "off" points sit on the other diagonal. No single straight line can put both diagonal corners of a square on one side while excluding the other diagonal — try to draw one and you'll find any line that separates (0,0) from (0,1) also cuts between (1,0) and (1,1) the wrong way, or vice versa. This isn't a limitation of a particular choice of weights; it is a geometric fact about the arrangement of the four points. XOR is not linearly separable, so no single neuron, however its weights are tuned, can compute it.

A common misconception is that the fix is simply "add more neurons in a single layer, side by side." It isn't — merely having more neurons doesn't help if they all look only at the raw inputs and their outputs are never combined by anything but another straight-line rule. What actually fixes XOR is depth: a hidden layer whose neurons each draw their own line, feeding into a further neuron that combines those lines' outputs. We can build this by hand before we ever write a training loop, which is the cleanest way to see why it works.

Solving XOR by hand with two lines

Look again at the diagram above. The two "light on" points both have x1 + x2 = 1. The "off" points have x1 + x2 equal to 0 or 2. So if we draw two parallel lines — one at x1 + x2 = 0.5 and one at x1 + x2 = 1.5 — the strip between them contains exactly the two "on" points and excludes both "off" points. Two lines, each drawable by one neuron, isolate what one line alone cannot.

Turn that into two hidden neurons, both using the threshold rule "fire if z ≥ 0":

• h1: weights (1, 1), bias -0.5 → fires whenever x1 + x2 ≥ 0.5 (this is OR)
• h2: weights (1, 1), bias -1.5 → fires whenever x1 + x2 ≥ 1.5 (this is AND)

Check all four inputs. For (0,0): h1 = 0, h2 = 0. For (0,1) and (1,0): h1 = 1, h2 = 0. For (1,1): h1 = 1, h2 = 1. Notice h1 and h2 only ever disagree — h1 = 1, h2 = 0 — exactly on the two inputs where the light is on. So a third neuron, taking h1 and h2 as its inputs, needs to fire only when h1 is high and h2 is low:

• output: weights (+1, -1) on (h1, h2), bias -0.5 → fires when h1 - h2 ≥ 0.5

Trace it: (0,0) → h1-h2 = 0-0 = 0, below 0.5, output 0. Correct. (0,1) and (1,0) → h1-h2 = 1-0 = 1, above 0.5, output 1. Correct. (1,1) → h1-h2 = 1-1 = 0, below 0.5, output 0. Correct. Six numbers — two weights and a bias, twice, plus one combining neuron — solve a problem that is provably impossible for any single neuron. This hand-built network is a genuine solution to XOR; nothing about it is approximate. The only thing missing is a way to find weights like these automatically, for problems where you can't just eyeball the geometry. That is what the rest of this chapter builds.

From hard switches to smooth neurons

The threshold rule ("fire if z ≥ 0, else don't") is easy to reason about by hand, but it is a terrible thing to learn from: it is flat everywhere except one point where it jumps discontinuously, so nudging a weight slightly almost never changes the output, and when it does, it changes it by a full jump. Learning algorithms need to feel a smooth slope guiding them toward better weights. So real networks replace the hard threshold with the sigmoid function:

sigmoid(z) = 1 / (1 + e^(-z))

Sigmoid squashes any real number into the open interval (0, 1), and it does so smoothly — for very negative z it's close to 0, for very positive z it's close to 1, and around z = 0 it rises steeply through 0.5. It behaves like a softened version of the threshold rule, but every point on it has a well-defined slope, which is exactly the handle a learning algorithm needs to grab onto. We'll use sigmoid for every neuron from here on.

The network architecture we'll train

We'll build a network with 2 input neurons (the switch positions), 4 hidden neurons (more than the 2 we needed by hand — a design choice explained below), and 1 output neuron, every hidden and output neuron using sigmoid. This is usually called a "2-4-1" network.

Every input connects to every hidden neuron, and every hidden neuron connects to the output — this is a fully connected (or "dense") layer, the default building block of neural networks. Each of the 4 hidden neurons has its own pair of weights and its own bias, so 8 weights and 4 biases connect the inputs to the hidden layer; the output neuron has 4 weights (one per hidden neuron) and 1 bias. In code, instead of writing eight separate multiplications, we store the 8 input-to-hidden weights as a 2×4 array W1 and let a single line compute all four hidden neurons' weighted sums at once. This bulk bookkeeping is a preview of matrix algebra, which you'll formalize in Class 12 — for now, treat X @ W1 as shorthand for "do all the weighted sums in one shot"; every value it produces is still just w1*x1 + w2*x2 + b, computed four times over.

import numpy as np

class TinyNet:
    def __init__(self, seed=0):
        rng = np.random.default_rng(seed)
        self.W1 = rng.normal(0, 1, (2, 4))   # input to hidden weights
        self.b1 = np.zeros((1, 4))
        self.W2 = rng.normal(0, 1, (4, 1))   # hidden to output weights
        self.b2 = np.zeros((1, 1))

    def sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def forward(self, X):
        self.z1 = X @ self.W1 + self.b1      # hidden pre-activations
        self.a1 = self.sigmoid(self.z1)      # hidden activations
        self.z2 = self.a1 @ self.W2 + self.b2  # output pre-activation
        self.a2 = self.sigmoid(self.z2)      # network output
        return self.a2

Weights start as small random numbers, not zeros. If every weight started at zero, every hidden neuron would receive exactly the same signal and compute exactly the same output at every step of training — they'd stay identical forever, collapsing 4 neurons into the learning power of 1. Random initialization breaks this symmetry so different hidden neurons can specialize into different lines, the way h1 and h2 specialized into OR and AND in our hand-built solution.

Measuring wrongness: the loss function

To improve the weights, the network first needs a number that says how wrong its current guess is. We'll use mean squared error:

def loss(self, X, y):
    y_hat = self.forward(X)
    return np.mean((y_hat - y) ** 2)

For each example, take the (prediction − true label), square it, then average across all 4 examples. Squaring does two things: it makes every error positive (an error of −0.3 hurts exactly as much as +0.3), and it punishes large errors more than proportionally — an error of 0.4 contributes 4 times as much loss as an error of 0.2, not just twice as much. Lower loss always means better predictions; a loss of 0 means every prediction exactly matches its label.

What a derivative actually measures

To reduce the loss, we need to know which direction to nudge each weight — up or down, and by how much. That's exactly what a derivative measures: the rate at which one quantity changes as another changes, at a single instant. You'll meet this formally in Class 11 (Limits and Derivatives) and build on it in Class 12; here's the working idea, built from scratch, that we need right now.

Take f(x) = x² and ask: how fast is f changing at x = 3? One way to estimate it is to see how much f changes over a small step h:

secant slope = (f(3+h) - f(3)) / h

This is just "rise over run" between two nearby points on the curve — literally the same slope formula from Class 10 coordinate geometry, applied to two points that happen to be very close together. Compute it for shrinking values of h:

h = 1      =>  ((4)^2 - 3^2) / 1     = (16 - 9)/1     = 7.000
h = 0.1    =>  (3.1^2 - 3^2) / 0.1   = (9.61 - 9)/0.1  = 6.100
h = 0.01   =>  (3.01^2 - 3^2)/0.01               = 6.010
h = 0.001  =>  (3.001^2 - 3^2)/0.001             = 6.001

As h shrinks toward 0, the slope converges to exactly 6 — which is 2 × 3. This isn't a coincidence: for f(x) = x², the slope at any point x converges to 2x (you'll prove this in general in Class 11 using limits). The derivative, written f'(x), is this limiting slope: the slope of the curve at a single point, found by zooming into an ever-shrinking secant.

Now apply the same zooming-in idea to sigmoid. It turns out — and we can check this numerically rather than prove it symbolically here — that sigmoid's derivative has a remarkably clean form:

sigmoid'(z) = sigmoid(z) * (1 - sigmoid(z))

Check it at z = 1: sigmoid(1) = 0.7311, so the formula predicts a slope of 0.7311 × (1 - 0.7311) = 0.1966. Estimating the slope directly the same way we did for x², with a tiny step of h = 0.0001, gives 0.1966 as well — matching to four decimal places. (You'll derive this formula symbolically in Class 12, using the quotient rule on 1/(1+e^-z); the numerical check confirms it's correct without requiring that machinery yet.)

The chain rule, built from a nudge

Our loss doesn't depend on a weight directly — it depends on the weight through a chain: the weight sets z, z sets the sigmoid output a, and a sets the loss. Here's the one-line intuition for how to combine sensitivities along a chain like this, which you'll meet formally as the chain rule in Class 12: if nudging w by a tiny amount moves z by (its own sensitivity) × (the nudge), and moving z moves a by (a's sensitivity to z) × (that shift), and moving a moves the loss by (loss's sensitivity to a) × (that shift), then the total effect of nudging w on the loss is the product of all three sensitivities, because each shift feeds directly into the next.

Let's verify this with real numbers instead of trusting it blindly. Take one neuron: input x = 2, weight w = 0.5, target 1, no bias.

z = w * x = 1.0
a = sigmoid(z) = 0.7311
L = (a - target)^2 = (0.7311 - 1)^2 = 0.0723

dL/da = 2*(a - target)        = 2*(0.7311 - 1) = -0.5379
da/dz = a*(1-a)               = 0.7311*0.2689  =  0.1966
dz/dw = x                     =  2.0000

chain rule: dL/dw = dL/da * da/dz * dz/dw = -0.5379 * 0.1966 * 2 = -0.2115

Now check this against a direct secant estimate of dL/dw at w = 0.5, using a tiny step exactly as we did for x²: it comes out to -0.2115 as well. The chain rule isn't a mysterious trick — it's just correctly bookkeeping three multiplied sensitivities, and we've confirmed the bookkeeping gives the right answer.

Backpropagation: applying the chain rule to every weight

Backpropagation is nothing more than this chain-rule bookkeeping applied systematically, layer by layer, starting from the loss and working backward toward the inputs — which is where the name comes from. For our network, the error signal at the output is dz2 = (y_hat - y) * y_hat * (1 - y_hat) — exactly the dL/da * da/dz product from above, just with the MSE derivative 2*(a-target) simplified (the constant 2 gets absorbed into the learning rate, a common simplification). That signal tells each hidden-to-output weight how to adjust: multiply the error signal by the hidden activation that fed into it. Then the same error signal, spread back through W2, becomes the error signal for the hidden layer, which in turn tells each input-to-hidden weight how to adjust:

def backward(self, X, y, lr=1.0):
    m = X.shape[0]
    y_hat = self.forward(X)

    dz2 = (y_hat - y) * y_hat * (1 - y_hat)      # output error signal
    dW2 = self.a1.T @ dz2 / m
    db2 = np.sum(dz2, axis=0, keepdims=True) / m

    da1 = dz2 @ self.W2.T                        # spread error back to hidden layer
    dz1 = da1 * self.a1 * (1 - self.a1)           # hidden error signal
    dW1 = X.T @ dz1 / m
    db1 = np.sum(dz1, axis=0, keepdims=True) / m

    self.W2 -= lr * dW2
    self.b2 -= lr * db2
    self.W1 -= lr * dW1
    self.b1 -= lr * db1

Every line here is the same three-term chain-rule product we hand-verified above, just applied to every weight in the network at once using the array bookkeeping from forward. lr, the learning rate, controls the size of each step — too large and the weights overshoot and oscillate; too small and training crawls.

Training on XOR and watching it learn

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([[0],[1],[1],[0]])

net = TinyNet(seed=0)
for epoch in range(5000):
    if epoch % 1000 == 0:
        print(f"epoch {epoch:5d}  loss = {net.loss(X,y):.4f}")
    net.backward(X, y, lr=1.0)

print(f"epoch  5000  loss = {net.loss(X,y):.4f}")
print(net.forward(X).round(3))

Running this exact code produces:

epoch     0  loss = 0.3430
epoch  1000  loss = 0.2188
epoch  2000  loss = 0.0233
epoch  3000  loss = 0.0061
epoch  4000  loss = 0.0033
epoch  5000  loss = 0.0022
[[0.02 ]
 [0.952]
 [0.952]
 [0.061]]

Read this honestly rather than expecting a smooth straight-line descent. For the first 1000 epochs the loss barely moves (0.343 → 0.219) — the random starting weights haven't yet found a useful direction, and each hidden neuron's sigmoid is sitting somewhere its slope is small, so the gradient signal is weak. Then, between epochs 1000 and 3000, the loss collapses by more than 30-fold as the hidden neurons' decision lines swing into positions resembling our hand-built OR/AND split. After that it settles into a slow final polish. The predictions after 5000 epochs — 0.02, 0.952, 0.952, 0.061 — round cleanly to the correct labels 0, 1, 1, 0, and the network reached this purely by following gradients; nobody told it to build an OR neuron and an AND neuron, but it effectively found a solution in that family.

This slow-start-then-fast-drop shape isn't guaranteed to look identical for every random seed, and gradient descent on this problem is not guaranteed to succeed at all: because the loss surface here isn't a simple bowl, a small number of unlucky starting weights can settle the network into a local minimum where the loss stalls well above zero and further training barely moves it. Using a few more hidden neurons than the bare minimum — we used 4 rather than the 2 our hand-built solution needed — gives the optimizer more alternative routes to a working solution and makes this considerably less likely, though not impossible.

Correcting a second misconception: depth needs nonlinearity to matter

It's tempting to think any multi-layer network can solve XOR just because it has multiple layers. That's false if the activation function is left out — algebra alone shows why. Suppose both layers were purely linear (no sigmoid, just z = w*x+b passed straight through):

a  = w1*x + b1                (hidden layer, linear)
out = w2*a + b2                (output layer, linear)
    = w2*(w1*x + b1) + b2
    = (w2*w1)*x + (w2*b1 + b2)

Expand the brackets and the two-layer network reduces to a single expression of the form (some number)*x + (some other number) — a straight line, indistinguishable from a single neuron with weight w2*w1 and bias w2*b1+b2. Stacking any number of purely linear layers, however many neurons wide, still only ever collapses back into one linear function, so it inherits exactly the same straight-line limitation we proved XOR defeats. The sigmoid in between the layers is not decoration — it's the one thing that stops the algebra from collapsing this way, since sigmoid(w1*x+b1) cannot be rewritten as w*x+b for any single w and b. Depth only buys expressive power when nonlinearity sits between the layers.

Where this connects to what you're studying now

Everything geometric in this chapter is Class 10 material used directly: a neuron's decision rule w1*x1+w2*x2+b=0 is a linear equation in two variables, and checking whether points satisfy an inequality like x1+x2 ≥ 0.5 is the same reasoning used for linear inequalities and their graphs. The slope calculation we used to build derivative intuition — (f(3+h)-f(3))/h — is literally the two-point slope formula from coordinate geometry, just applied to points that get closer and closer together. The calculus ideas — derivatives as limiting slopes, and the chain rule for composed functions — are previews: you'll build them rigorously in Class 11's Limits and Derivatives and Class 12's Continuity and Differentiability, and later, if you pursue engineering entrance exams or a computing degree, you'll use exactly this backpropagation machinery again in more general form. Nothing here needs those tools taken on faith today — every derivative claim in this chapter was checked numerically, not just asserted.

Check your understanding

1. Why can one neuron with a threshold or sigmoid activation compute AND and OR, but not XOR? Answer using the straight-line argument, not memorized rules.
2. In the hand-built solution, what would happen to the output neuron's decision if h1's bias were changed from -0.5 to -0.9? Recompute the truth table for h1 and check whether the whole network still outputs the correct XOR values.
3. Using the secant-slope method demonstrated for f(x)=x², estimate the derivative of f(x)=x² at x=5 using h=0.01, and check it against 2x.
4. In the backward function, da1 = dz2 @ self.W2.T spreads the output error back to the hidden layer. Explain in one sentence, using the chain-rule idea, why this step needs W2 and not W1.
5. If every weight in TinyNet were initialized to exactly 0 instead of a random value, what would h1, h2, h3, h4 compute after one training step, and why would this stay true forever? Refer back to the symmetry-breaking argument.

Summary

A single neuron's decision boundary is a straight line, so it can only solve linearly separable problems like AND and OR — never XOR, whose "on" and "off" points sit on opposite diagonals of the input square. Stacking a hidden layer of neurons, each drawing its own line, and combining their outputs in a further neuron solves XOR exactly, as we verified by hand with six numbers before writing any training code. Real networks replace the hard threshold with the smooth sigmoid function so that a derivative — the limiting slope of secants, exactly like the two-point slope formula from coordinate geometry — exists everywhere and can guide learning. The chain rule lets that guidance flow backward through every layer as a product of local sensitivities; backpropagation is this chain-rule bookkeeping applied to every weight in the network simultaneously, using array notation purely as a shortcut for many ordinary weighted sums done at once. Training on XOR with this exact code showed a real, sometimes uneven trajectory — a slow start, a sharp drop, a long polish — landing at predictions of 0.02, 0.952, 0.952, and 0.061 against true labels 0, 1, 1, 0. And critically, none of this works without nonlinearity between the layers: stack purely linear layers and the algebra collapses them back into a single straight line, no matter how many you stack.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where building a neural network from scratch in python is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting building a neural network from scratch in python to at least 3 other topics you have studied.