Convolutional Neural Networks: How Computers See

Open the PhonePe or Google Pay camera, point it at a UPI QR code, and it reads the code in under a second even if the sticker is creased, tilted, or half in shadow. Open a banking app and it verifies your face against your Aadhaar photo for e-KYC. Somewhere in the ISRO data-processing pipeline, software scans thousands of Chandrayaan orbital images looking for crater-shaped patterns too numerous for any human team to mark by hand. All three systems are solving the same underlying problem: turning a grid of pixel numbers into a correct label, reliably, no matter where in the frame the important pattern sits. The architecture that made this practical is the convolutional neural network, or CNN. This chapter builds one from first principles — not by naming its parts, but by deriving why each part has to exist.

Why a Plain Neural Network Fails on Images

Suppose you already know the basic feedforward neural network: layers of neurons, each connected to every neuron in the previous layer, weights learned by gradient descent. It is natural to ask why we don't just flatten an image into a long list of pixel values and feed it into one of these "dense" networks. Try the arithmetic and the answer becomes obvious.

A modest colour photograph, say 224×224 pixels with 3 colour channels (red, green, blue), has 224 × 224 × 3 = 150,528 numbers. Connect that flattened vector to a first hidden layer of just 1,000 neurons, and the weight count for that single layer alone is:

150,528 inputs × 1,000 neurons = 150,528,000 weights
(+ 1,000 bias terms ≈ 150.5 million parameters, first layer only)

That is before you've added a second layer, before you've trained on enough images to fit 150 million free parameters without wildly overfitting, and before you've solved the deeper problem: a dense layer treats pixel (0,0) and pixel (223,223) as having nothing to do with each other. If the network learns to recognise a cat's eye when it appears in the top-left corner during training, it has learned nothing about recognising that same eye if it appears in the bottom-right corner of a different photo. Every neuron has its own private set of weights for every pixel position, so the network must re-learn every visual pattern separately at every possible location. This is both computationally wasteful and statistically absurd — the physics of what makes something "look like an eye" does not change depending on where in the photo it happens to sit.

CNNs fix both problems with one idea: instead of connecting every neuron to every pixel, use small filters that slide across the entire image, reusing the same weights at every position.

The Core Idea: A Small Pattern-Detector That Slides

Picture a jaali (a perforated stone or wood screen, common in Mughal and Rajasthani architecture) laid on top of a photograph, small enough to cover only a 3×3 patch of pixels at a time. You slide it across the photo, and at every position you ask one question: "how strongly does the pattern under this window match the pattern the jaali is looking for?" You get a number back at every position. Slide it across the whole image and you've produced a new, smaller grid of "match strength" scores — high wherever the pattern was present, low wherever it wasn't.

That sliding, pattern-matching window is a convolutional filter (also called a kernel), and the grid of match-strength scores it produces is a feature map. The filter's weights are not hand-designed — they are learned by gradient descent, exactly like the weights in a dense layer. What's different is that the same 3×3 (or 5×5, or 7×7) set of weights is reused at every single position in the image. This reuse is called parameter sharing, and it is the single most important design decision in a CNN: it encodes the assumption that a useful pattern — an edge, a curve, a corner — is worth detecting the same way no matter where it appears.

The Convolution Operation, Defined Precisely

Let an input image be a matrix I of size n × n, and a filter be a smaller matrix K of size f × f. The output at position (i, j) is the sum of the element-wise product between the filter and the patch of the image it currently sits on:

output[i, j] = sum over (a, b) of I[i+a, j+b] * K[a, b]
               for a = 0 .. f-1, b = 0 .. f-1

In words: overlay the filter on a patch of the image the same size as the filter, multiply each overlapping pair of numbers, and add up all the products into one number. That one number becomes one pixel of the output feature map. Then slide the filter one step over and repeat.

Let's do this by hand, because the arithmetic is where the intuition actually lands. Take this 5×5 input (think of it as a tiny patch of a grayscale image, values 0–3 for simplicity):

I =
1 2 3 0 1
0 1 2 3 1
1 0 1 2 0
2 1 0 1 3
0 2 1 0 1

and this 3×3 filter:

K =
 1  0 -1
 1  0 -1
 1  0 -1

Notice the middle column of K is entirely zero. That's deliberate: this filter ignores the centre of every patch and computes (sum of the left column) minus (sum of the right column). If the left side of a patch is brighter than the right side, the output is a large positive number; if the right side is brighter, the output is negative; if both sides are equal, the output is zero. This filter is a vertical edge detector — it fires on brightness changes that run left-to-right, which is exactly what a vertical edge in the image looks like.

Now compute the top-left output value by hand. The filter sits over the top-left 3×3 patch of I:

patch =
1 2 3
0 1 2
1 0 1

Multiply element-by-element with K and sum:

(1×1 + 2×0 + 3×(-1))
+ (0×1 + 1×0 + 2×(-1))
+ (1×1 + 0×0 + 1×(-1))
= (1 + 0 - 3) + (0 + 0 - 2) + (1 + 0 - 1)
= -2 + -2 + 0
= -4

That -4 becomes the top-left entry of the output. Slide the filter one column to the right and repeat over the next patch, and so on across all valid positions. The SVG below shows exactly this step: the highlighted patch in the input, the filter, and the resulting output cell.

If you slide the filter across all nine valid positions and repeat this arithmetic each time, the full output feature map turns out to be:

-4 -2  4
 0 -4 -1
 1  0 -2

You can check this yourself with a shortcut once you notice the filter's middle column is zero: at each position, the output is just (sum of the patch's left column) minus (sum of the patch's right column). For the top-right patch, columns 2 and 4 of I restricted to rows 0–2 give left-column sum 3+2+1=6 and right-column sum 1+1+0=2, so output = 6-2 = 4 — matching the grid above. This filter is genuinely detecting where brightness jumps from low (left) to high (right) or vice versa within each patch: a real, working vertical-edge detector, arrived at with nothing more than 18 multiplications.

Verifying It in Code

Here is the exact same computation as a NumPy function, so you can confirm the hand arithmetic is not a trick:

import numpy as np

def conv2d(image, kernel):
    ih, iw = image.shape
    kh, kw = kernel.shape
    oh, ow = ih - kh + 1, iw - kw + 1
    output = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            patch = image[i:i+kh, j:j+kw]
            output[i, j] = np.sum(patch * kernel)
    return output

I = np.array([
    [1, 2, 3, 0, 1],
    [0, 1, 2, 3, 1],
    [1, 0, 1, 2, 0],
    [2, 1, 0, 1, 3],
    [0, 2, 1, 0, 1],
])

K = np.array([
    [1, 0, -1],
    [1, 0, -1],
    [1, 0, -1],
])

print(conv2d(I, K))
# [[-4. -2.  4.]
#  [ 0. -4. -1.]
#  [ 1.  0. -2.]]

Trace it: oh = ow = 5 - 3 + 1 = 3, so the loop runs i, j from 0 to 2, nine times. On the first iteration (i=0, j=0), patch is image[0:3, 0:3], the same 3×3 block used above, and np.sum(patch * kernel) performs the element-wise multiply-and-add that gives -4. The loop then fills every other cell the same way, matching the hand-computed grid exactly.

Why "Convolution" Is a Slightly Misleading Name

Here is a genuine, commonly missed technicality worth correcting explicitly. In signal processing, mathematical convolution requires flipping the kernel 180° before sliding it — that flip is part of the formal definition, and it's what makes convolution obey nice algebraic properties like commutativity. What CNNs actually compute, as in the operation above, is cross-correlation: slide and multiply, no flip. Deep learning frameworks call the layer "Conv2D" anyway, and the distinction changes nothing about how the network learns, because the filter's weights are learned from scratch by gradient descent — if flipping mattered, the optimiser would simply learn the flipped weights instead. But if you ever compare a hand-derivation like the one above against a strict signal-processing textbook and the numbers don't match, this is why: you're looking at cross-correlation, and that is what every CNN in production actually uses.

Many Filters, Not Just One

A single filter can only detect one kind of pattern. A real convolutional layer applies many filters — commonly 32, 64, or more — to the same input, each with its own independently learned weights, each producing its own feature map. Stack all those feature maps together and you get a 3D block of output: (height) × (width) × (number of filters). This is why a "channel" count grows as you go deeper into a CNN even as the height and width shrink: you start with 3 colour channels and end up with dozens or hundreds of learned "feature channels," each tuned to a different kind of visual pattern — one might respond to vertical edges, another to a particular shade of green, another (much deeper in the network) to something as specific as "a rounded shape with two dark ovals," which starts to resemble an eye detector purely as an emergent consequence of training, not because anyone programmed it to look for eyes.

Now redo the parameter count from the introduction, but with a real convolutional layer. Take a 224×224×3 input and a conv layer with 64 filters, each 3×3×3 (3×3 spatially, spanning all 3 input channels):

Weights per filter = 3 × 3 × 3 = 27
Total weights       = 27 × 64 = 1,728
Total (+ 1 bias per filter) = 1,728 + 64 = 1,792 parameters

Compare: 1,792 parameters for the convolutional layer against roughly 150.5 million for the dense layer covering the same input — about 84,000 times fewer. And crucially, those 1,792 numbers work identically no matter where in the 224×224 image the pattern shows up, because the same 64 filters are reused at every position by sliding.

Padding and Stride: Controlling the Output Size

Notice that convolution shrinks the image: a 5×5 input with a 3×3 filter produced only a 3×3 output. Applied repeatedly across many layers, an image would shrink to nothing before you got very deep. Two knobs control this.

Padding (P) adds a border of zeros around the input before sliding the filter, so the output can stay the same size as the input if desired ("same" padding) instead of shrinking ("valid" padding, P=0).

Stride (S) is how many pixels the filter jumps between positions. Stride 1 slides one pixel at a time (as in the worked example above); stride 2 skips every other position, halving the output resolution and reducing computation.

Putting both together, for an n × n input, an f × f filter, padding P, and stride S, the output size is:

O = floor( (n + 2P - f) / S ) + 1

Derivation: padding adds P zeros to each side, so the effective input width becomes n + 2P. The filter's left edge can start at position 0 and must stop once its right edge would run past the effective input, i.e. the last valid start position is (n + 2P) - f. The number of valid starting positions, stepping by S each time starting from 0, is floor(((n + 2P) - f) / S) + 1 — the "+1" accounts for position 0 itself. Check it against the worked example: n=5, f=3, P=0, S=1 gives O = floor((5-3)/1)+1 = 3, matching the 3×3 output computed above.

ReLU: Why a Nonlinearity Has to Sit Between Convolutions

A convolution is a linear operation — it's just weighted sums. Stack two linear layers directly on top of each other with nothing in between and the composition is still just one linear operation; you gain no extra representational power, only extra parameters. So exactly as in a dense network, every convolutional layer is followed by a nonlinear activation function, almost always the Rectified Linear Unit: ReLU(x) = max(0, x). Applied to the feature map computed above, every negative entry is clipped to zero and every positive entry passes through unchanged:

Before ReLU:        After ReLU:
-4 -2  4              0  0  4
 0 -4 -1     ==>      0  0  0
 1  0 -2               1  0  0

Only the two patches where the vertical-edge filter fired strongly positive survive; the rest are silenced to zero. This sparsity is not incidental — it's a large part of why deep CNNs remain trainable and computationally cheap: at any given layer, most neurons in most feature maps are exactly zero for a typical input.

Pooling: Downsampling on Purpose

After a convolution+ReLU stage, CNNs typically apply a pooling layer, most commonly max pooling: slide a small window (usually 2×2) across the feature map with stride equal to the window size, and keep only the maximum value in each window, discarding the rest. Take this 4×4 feature map:

3 1 2 4
5 6 1 2
1 2 0 1
3 4 2 1

A 2×2 max-pool with stride 2 splits it into four non-overlapping 2×2 blocks — top-left {3,1,5,6}, top-right {2,4,1,2}, bottom-left {1,2,3,4}, bottom-right {0,1,2,1} — and keeps each block's maximum:

6 4
4 2

Two things happen here. First, dimensionality drops by a factor of 4 (16 numbers become 4), which cuts computation in every subsequent layer. Second, and more subtly, pooling buys a small amount of translation tolerance: if the strong response in a block shifts by one pixel due to the underlying pattern in the photo moving slightly, the maximum of that block is often unchanged, so the network's output is more stable to small shifts of the input. Pooling is common but not mandatory — many modern architectures (ResNet and its descendants, for instance) replace some pooling layers with a stride-2 convolution instead, which downsamples while also learning a useful filter at the same time. It's worth naming this as a genuine misconception: pooling is not a required ingredient of a CNN, it's one of at least two standard ways to downsample, and current research more often favours learned strided convolutions.

Stacking Layers: Why Depth Builds Hierarchy, and the Receptive Field

A single 3×3 convolution only "sees" a 3×3 window of the original image — its receptive field is 3×3. But feed the output of one conv layer into another 3×3 conv layer, and each unit in the second layer's output is computed from a 3×3 patch of the first layer's output — which itself was computed from a 3×3 patch of the original image around each of those points. Work through the geometry and, for stride-1 layers, the receptive field grows by (f - 1) pixels per additional layer:

RF(L layers of f×f, stride 1) = 1 + L × (f - 1)

For three stacked 3×3 layers: RF = 1 + 3×2 = 7 — the same receptive field as a single 7×7 convolution, but using 3 × (3×3) = 27 weights per channel instead of 7×7 = 49, and with three ReLU nonlinearities injected in between instead of one. This is the actual argument used to justify stacking small filters instead of using large ones directly — it was central to how VGGNet, a well-known 2014 image-classification architecture, was designed. The practical consequence for what a CNN learns is a hierarchy: early layers, with small receptive fields, learn simple local patterns like edges and colour gradients (as the hand-computed filter above did); middle layers, with medium receptive fields built from combinations of early features, learn textures and simple shapes; deep layers, with receptive fields spanning most or all of the image, learn parts and whole objects. Nobody hand-designs this hierarchy — it emerges from stacking the same simple operation (convolve, ReLU, sometimes pool) and training end-to-end with backpropagation, and it has been directly confirmed by visualisation studies that reconstruct what pattern makes each deep-layer unit fire most strongly.

Worked Example: A Small Digit-Recognition Network

India Post's automated PIN-code sorting problem — reading a handwritten 6-digit PIN off an envelope and routing it correctly — is the same category of problem that motivated some of the earliest successful CNNs: LeCun's late-1990s networks were trained to read handwritten digits on postal mail and bank cheques. Here is a compact CNN in that spirit, sized for a 28×28 grayscale digit image:

Input:            28 × 28 × 1
Conv1  (6 filters, 5×5, stride 1, no padding):
                   output 24×24×6   (28-5+1=24)
ReLU
MaxPool (2×2, stride 2):
                   output 12×12×6
Conv2  (16 filters, 5×5, stride 1, no padding):
                   output  8× 8×16  (12-5+1=8)
ReLU
MaxPool (2×2, stride 2):
                   output  4× 4×16  = 256 values, flattened
Dense (256 -> 120) -> ReLU
Dense (120 -> 84)  -> ReLU
Dense (84  -> 10)  -> Softmax over 10 digit classes

Count the parameters layer by layer: Conv1 has 5×5×1×6 = 150 weights + 6 biases = 156. Conv2 has 5×5×6×16 = 2,400 weights + 16 biases = 2,416. Dense(256→120) has 256×120 = 30,720 weights + 120 biases = 30,840. Dense(120→84) has 120×84=10,080 + 84 = 10,164. Dense(84→10) has 84×10=840 + 10 = 850. Total: 156 + 2,416 + 30,840 + 10,164 + 850 = 44,426 parameters — a network you could genuinely train on a laptop, and small enough to run inference on a low-power scanner at a sorting facility, in contrast to the 150-million-parameter estimate for a single dense layer on a much larger image at the start of this chapter. Note also where most of the parameters live: not in the convolutional layers (2,572 combined) but in the dense layers after flattening (41,854 combined) — a pattern that holds in essentially every classic CNN, and one reason later architectures work hard to reduce or eliminate large dense layers near the output.

Where This Sits in Your Exams

CNNs are not part of the JEE Main/Advanced syllabus, which stays within physics, chemistry, and mathematics — don't spend scarce JEE-prep time here expecting board-adjacent payoff. Where this material does count directly: CBSE's Artificial Intelligence subject (code 417, offered as a skill subject from Class 9 onward) includes neural-network and deep-learning concepts at exactly this level of rigour, and GATE's Data Science and Artificial Intelligence paper (introduced in 2024 as a full GATE subject) tests CNN architecture and computation directly — the parameter-counting and output-size derivations above are precisely GATE-DA style questions. If competitive breadth beyond the board interests you, the International Olympiad in Artificial Intelligence (IOAI), which held its first edition in 2025, includes exactly this kind of applied deep-learning reasoning.

Check Your Understanding

• A 7×7 grayscale image is convolved with a 3×3 filter, stride 1, no padding. What are the output dimensions? Now redo it with padding 1. Now redo it with stride 2 and padding 1.
• Design a 3×3 filter (by choosing 9 numbers) that would respond strongly to a horizontal edge (a boundary between a bright row above and a dark row below) rather than the vertical-edge filter used in this chapter's worked example. Explain your reasoning before checking by computing one patch by hand.
• A colleague says, "we don't need pooling — the convolution already shrinks the image, so that's the same thing." Explain precisely what is wrong with this claim, using the distinction between what shrinks the output (stride, filter size, padding) and what pooling does that plain convolution does not.
• A conv layer takes a 32×32×3 input and applies 128 filters of size 5×5. How many learnable parameters does this one layer have? Compare that to a dense layer connecting the same flattened 32×32×3 input to 128 output neurons, and state the ratio.
• Explain, in your own words and without using the word "convolution," why reusing the same filter weights at every image position is a reasonable assumption to bake into the architecture, and describe one type of image where that assumption would be a poor fit.

Summary

A convolutional neural network replaces the dense layer's "one private weight per pixel per neuron" with a small filter that slides across the whole image, reusing the same weights everywhere — cutting parameter counts by orders of magnitude and encoding the assumption that a useful visual pattern means the same thing regardless of where it appears. Each filter performs cross-correlation (multiply-and-sum over a sliding window) to produce a feature map; padding and stride control how the output size shrinks, following O = floor((n + 2P - f)/S) + 1; a nonlinearity (ReLU) after every convolution is what keeps stacked layers from collapsing into one linear operation; pooling (or a strided convolution) downsamples while adding a degree of translation tolerance; and stacking many such layers grows the receptive field layer by layer, which is why deep CNNs learn a genuine hierarchy — edges, then textures and parts, then whole objects — without that hierarchy ever being hand-specified.

Think About It

Think about this: How would you explain convolutional neural networks: how computers see to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where convolutional neural networks: how computers see is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting convolutional neural networks: how computers see to at least 3 other topics you have studied.