Convolutional Neural Networks for Images

Open the Photos app on your phone and type "dog" into the search bar. In a fraction of a second it pulls up every picture of a dog you have ever taken — puppies at a wedding in Pune, a street dog near your school, your neighbour's Labrador. Nobody sat down and tagged those photos. So how does the phone know what a dog looks like? It was never given a rule like "a dog has four legs and a tail," because a cat has those too, and a dog curled up asleep might show no legs at all. The machine learned to recognise dogs the way you did as a toddler: by looking at thousands of examples until the visual pattern of "dog" became familiar. The technology that made this possible — and that today reads handwriting on cheques, spots tumours in X-rays, and lets a self-driving car see a pedestrian — is the Convolutional Neural Network, or CNN. This chapter builds one up from a single, very simple idea: sliding a tiny stencil across an image.

First, what is an image to a computer?

Before a machine can recognise anything, it has to store the picture as numbers, because numbers are the only thing a computer truly holds. A grayscale (black-and-white) image is a grid of numbers, where each number is the brightness of one tiny square called a pixel. By convention 0 means pure black and 255 means pure white, with the greys in between. A colour image is three such grids stacked together — one for Red, one for Green, one for Blue — which is why we say a colour image has three channels.

Imagine a small 5×5 grayscale image of a bright square sitting in a darker background. As numbers it might look like this:

  0   0   0   0   0
  0 200 200 200   0
  0 200 200 200   0
  0 200 200 200   0
  0   0   0   0   0

Your eye instantly sees a white block in the middle. The computer sees only twenty-five numbers. The entire job of a CNN is to turn those raw numbers into a meaningful answer like "square" or "dog" or "the digit 7" — and it does this by hunting for patterns in the grid.

The core idea: a sliding stencil called a filter

Here is the single most important idea in this whole chapter. Instead of looking at the whole picture at once, a CNN looks at a small window at a time — say a 2×2 or 3×3 patch — and asks one narrow question about that patch: "Does the pattern I care about appear here?" The little grid of numbers that encodes the question is called a filter (also called a kernel). We slide this filter across every position in the image, and at each stop we compute a single number that says how strongly the filter's pattern matches that spot. Sliding a filter across an image like this is the operation called convolution. That is literally where the name Convolutional Neural Network comes from.

Think of the filter as a small transparent stencil you drag across a photo. When the shape cut into the stencil lines up with something in the photo, a bell rings loudly; when it does not line up, the bell stays quiet. The louder the bell across the image, the more that pattern is present, and where.

How does the "bell" produce a number? At each window we multiply each filter number by the pixel number sitting underneath it, then add up all those products into one total. That single total is the filter's response at that location. Let us do it slowly with real numbers, because this arithmetic is the beating heart of a CNN and every student should be able to do it by hand.

A fully worked convolution — detecting a vertical edge

Edges — places where brightness jumps from light to dark — are the building blocks of everything we see. Let us build a filter that detects a vertical edge, meaning "bright on the left, dark on the right." A natural choice is this 2×2 filter:

filter = [[ 1, -1],
          [ 1, -1]]

Read it as: reward brightness on the left column (+1), punish brightness on the right column (−1). Now take a small image region that is bright on the left and dark on the right — exactly the edge we want to find:

region = [[10, 10, 0],
          [10, 10, 0],
          [10, 10, 0]]

The filter is 2×2 and the region is 3×3, so the filter fits into four different positions as it slides. Let us compute all four. At each position we line the filter up on top of a 2×2 slice, multiply matching cells, and sum.

Position 1 — top-left, covering rows 0–1, columns 0–1. The slice underneath is [[10,10],[10,10]]:

(1 × 10) + (-1 × 10) + (1 × 10) + (-1 × 10)
= 10 - 10 + 10 - 10 = 0

The result is 0. That makes sense: this window is entirely inside the bright area, so there is no edge here — nothing changes from left to right, so the "bell" stays silent.

Position 2 — top-right, columns 1–2. The slice is [[10,0],[10,0]]:

(1 × 10) + (-1 × 0) + (1 × 10) + (-1 × 0)
= 10 - 0 + 10 - 0 = 20

The result is 20 — a loud response! This window straddles the boundary where bright (10) meets dark (0), which is precisely the vertical edge the filter was built to find. The two lower windows (rows 1–2) give the identical pattern, so the full output grid is:

output = [[ 0, 20],
          [ 0, 20]]

Look at what happened: the filter turned a raw block of pixels into a clean map that says "there is a strong vertical edge along the right side, and nothing on the left." The high numbers point exactly at where the edge lives. This little numeric result is called a feature map — a map of where a particular feature (here, a vertical edge) was found. This is the entire magic of convolution, and everything else in a CNN is built on top of it.

Here is a diagram of the filter sliding across the image and producing the feature map:

Why sliding one small filter is such a clever trick

You might wonder: why not just connect every pixel to every decision, the way an ordinary neural network would? Two reasons, and both matter enormously.

Reason 1 — parameter sharing. A vertical edge is a vertical edge whether it appears in the top-left of the photo or the bottom-right. So we use the same four filter numbers everywhere as we slide. Compare the counting: a modest 100×100 image has 10,000 pixels. A traditional "fully connected" layer with just 100 detectors would need 10,000 × 100 = one million weights. A CNN detector that scans the whole image needs only the handful of numbers in its filter — for a 3×3 filter, just 9. That is why CNNs can be trained on ordinary hardware and even run offline on a small device like a Raspberry Pi.

Reason 2 — translation invariance. Because the same filter checks every location, a CNN can find a cat's ear whether the cat is centred or off to the side. A network that memorised "cat ear belongs at pixel (40, 12)" would fail the moment the cat moved. Sharing the filter across positions bakes in the common-sense idea that a pattern means the same thing wherever it appears.

And crucially, the filter numbers are not written by a human. We only decide how big the filters are and how many there are. The actual values — the +1s and −1s — are learned automatically during training, by showing the network labelled examples and nudging the numbers whenever it guesses wrong. Early in training the filters are random noise; after training, they settle into edge-detectors, corner-detectors, colour-blob detectors, and so on, entirely on their own.

Common misconception: "the filter is a small picture the network searches for"

Many students first imagine that a filter is a tiny thumbnail photo — a mini-dog — that the computer literally hunts for inside the big image, like a game of spot-the-picture. That is not what happens, and believing it will confuse you later. A filter is not a picture; it is a small grid of weights that scores a pattern of brightness change. Our edge filter above contains negative numbers — you cannot have a "negative brightness" in a real photo, so it is clearly not an image. It is a question. And a single filter never detects a whole dog. The first layer's filters detect only the humblest features: edges, corners, small colour patches. It takes many stacked layers before those crumbs combine into anything as complex as "dog." Which brings us to depth.

Stacking layers: from edges to eyes to dogs

One convolution layer applies many filters at once — say 16 or 32 different filters — producing 16 or 32 feature maps, each highlighting one kind of simple pattern. We then feed those feature maps into a second convolution layer, whose filters now combine simple patterns into slightly bigger ones. A layer that sees "a horizontal edge above a vertical edge meeting at a point" has effectively learned a corner. A still deeper layer might combine two curves and a dark blob into "an eye." The deepest layers assemble eyes, ears, and fur textures into "dog." This ladder — edges → textures → parts → objects — is the reason CNNs are built deep, and it mirrors how the human visual brain is organised, from simple cells that fire at edges to higher regions that recognise faces.

Two more essential ingredients: ReLU and pooling

Between convolution layers, a CNN does two small but vital extra steps.

The activation function (ReLU). After computing a feature map we pass every number through a rule called ReLU (Rectified Linear Unit), which is refreshingly simple: if the number is negative, make it 0; otherwise keep it. So −7 becomes 0, and 20 stays 20. Why bother? Without a rule like this, stacking many layers would just be adding and multiplying in straight lines, which mathematically collapses into a single simple layer no matter how deep you go — you would gain nothing from depth. ReLU introduces a "bend" (a non-linearity) that lets deep networks actually learn rich, curvy patterns. It also throws away "negative evidence," keeping only the places where a feature is genuinely present.

Pooling. Feature maps are large, and we do not need pixel-perfect precision about where an edge is — roughly where is enough. Max pooling shrinks a feature map by sliding a small window (usually 2×2) and keeping only the largest value in each window. Consider this 4×4 feature map pooled with a 2×2 window that jumps 2 steps at a time (we call that jump the stride):

feature map (4×4)          after 2×2 max pool (2×2)
[[1,  3,  2,  1],
 [4,  6,  1,  0],           top-left window max(1,3,4,6)=6
 [2,  1,  0,  8],    →      [[6, 2],
 [5,  2,  3,  4]]            [5, 8]]

Trace it: the top-left 2×2 block {1,3,4,6} → 6; top-right {2,1,1,0} → 2; bottom-left {2,1,5,2} → 5; bottom-right {0,8,3,4} → 8. The 4×4 grid becomes a 2×2 grid — four times smaller — while keeping the strongest signal in each region. Pooling makes the network faster, uses less memory, and makes it robust: if the cat shifts by one pixel, the pooled answer barely changes.

How big is the output? A formula you can compute

When a filter slides over an image, the output feature map is smaller than the input, because the filter cannot hang off the edges. The size follows a clean formula. If the input is N pixels wide, the filter is F wide, and the stride (jump size) is S, then the output width is:

output size = (N - F) / S + 1

Let us check it against our very first edge example: the region was N = 3 wide, the filter F = 2 wide, stride S = 1. So (3 − 2)/1 + 1 = 2. And indeed our feature map was 2 wide. It works. Try a bigger one: a 5×5 image with a 3×3 filter at stride 1 gives (5 − 3)/1 + 1 = 3, a 3×3 output. With stride 2 instead: (5 − 3)/2 + 1 = 2. Bigger strides shrink the output faster because the filter takes bigger jumps. (Engineers sometimes add a border of zeros around the image, called padding, so the output stays the same size as the input — but the formula above is the foundation you need first.)

Putting it all together: a CNN in code

Here is the shape of a complete small CNN written in Python-like pseudocode, the kind used to recognise handwritten digits (say, reading the pincode on a letter for India Post). Read it top to bottom as the picture's journey:

model = Sequential()

# Input: a 28×28 grayscale image (1 channel)
model.add(Conv2D(filters=16, size=3, activation='relu'))  # find 16 edge/texture patterns
model.add(MaxPool2D(size=2))                               # shrink, keep strongest signals
model.add(Conv2D(filters=32, size=3, activation='relu'))  # combine into bigger patterns
model.add(MaxPool2D(size=2))                               # shrink again
model.add(Flatten())                                       # unroll the grid into a list
model.add(Dense(units=10, activation='softmax'))           # 10 outputs: digits 0–9

# The final layer outputs 10 probabilities that add up to 1.
# For an image of a '7', a trained network might output:
#   [0.01, 0.00, 0.02, 0.01, 0.00, 0.01, 0.00, 0.93, 0.01, 0.01]
# The largest is at position 7 (0.93), so the network answers: "7".

Notice the overall pattern: convolve → ReLU → pool, repeated to build up complexity, then flatten the final grid into a plain list and feed it to an ordinary decision layer that outputs one probability per class. The softmax at the end turns the raw scores into probabilities that sum to 1, so we can read the biggest one as the network's confident answer. This exact architecture, trained on 60,000 example digits, reaches well above 98% accuracy — good enough to sort mail automatically.

Where you meet CNNs in real life

CNNs are quietly everywhere around you in India. When you deposit a cheque through a banking app, a CNN reads the handwritten amount. When a UPI app scans a QR code, edge-detecting filters like the one we built find the code's black-and-white squares. Radiologists in hospitals increasingly use CNN tools to flag suspicious spots in chest X-rays for tuberculosis. Agricultural apps let a farmer photograph a diseased crop leaf and get a diagnosis, because a CNN was trained on thousands of leaf images. And ISRO uses CNN-based systems on satellite imagery to map floods and monitor crops across the country. Every one of these rests on the same humble operation you traced by hand: slide a small filter, multiply, and add.

Active recall — do these yourself

1. Convolve by hand. Take the filter [[1, −1]] (a 1×2 horizontal edge detector) and slide it across the single row [5, 5, 9, 9]. Compute all three output values. (Answer: (5−5)=0, (5−9)=−4, (9−9)=0 → [0, −4, 0]. The edge shows up where 5 meets 9.)
2. Apply ReLU. Pass your answer [0, −4, 0] through ReLU. What do you get, and why did the −4 disappear? (Answer: [0, 0, 0]. ReLU zeros negatives; this filter fired for a dark-to-bright edge, but ReLU keeps only bright-to-dark evidence. Flip the filter sign to catch the other direction.)
3. Output size. An image is 32 pixels wide. A filter is 5 wide, stride 1. How wide is the feature map? Now recompute with stride 3. (Answers: (32−5)/1+1 = 28; (32−5)/3+1 = 10.)
4. Max pool. Pool this 2×2 map with a single 2×2 window: [[7, 2], [1, 9]]. What is the result? (Answer: 9.)
5. Explain in words. Your friend says "a CNN just searches the image for a small photo of the object." Write two sentences correcting this, using the idea that a filter is a grid of weights (including negatives) that scores a pattern of brightness change, not a picture.
6. Count the savings. A 3×3 filter has how many weights? A fully connected detector on a 50×50 image has how many? Which is easier to train, and why does parameter sharing cause the gap? (Answers: 9 versus 2,500; the filter, because the same 9 weights are reused at every position.)

Summary — the key ideas to keep

• An image is just a grid of pixel-brightness numbers (three grids for colour: R, G, B).
• Convolution slides a small filter (a grid of weights) across the image; at each stop it multiplies overlapping numbers and sums them into one value, building a feature map that shows where the filter's pattern appears.
• Filters are learned, not hand-written; the same filter is reused at every position (parameter sharing), which is why CNNs need so few weights and recognise a pattern wherever it appears (translation invariance).
• ReLU zeros out negative values, adding the non-linear "bend" that makes depth worthwhile. Max pooling shrinks feature maps by keeping the strongest value in each window, saving memory and adding robustness.
• Stacking layers builds a hierarchy: edges → textures → parts → whole objects.
• Output width follows (N − F)/S + 1; the network ends by flattening the final grid and using softmax to output one probability per class.
• A filter is a question about brightness change, not a tiny picture the computer hunts for — the negative weights alone prove it.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where convolutional neural networks for images is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting convolutional neural networks for images to at least 3 other topics you have studied.