Image Classification: Teaching Machines to See

What a Computer Actually Receives When You Show It a Photo

Look at these two grids of digits before reading any further. One represents the handwritten numeral "1" and the other represents "7." Both use a scale from 0 (white paper) to 9 (dark ink).

Grid A:            Grid B:
0 0 9 0 0           9 9 9 9 9
0 0 9 0 0           0 0 0 9 0
0 0 9 0 0           0 0 9 0 0
0 0 9 0 0           0 9 0 0 0
0 0 9 0 0           9 0 0 0 0

You can probably tell that Grid A is the "1" (a single vertical stroke) and Grid B is the "7" (a horizontal top bar with a diagonal stroke falling away to the left). Now here is the point of the exercise: this is not a simplified representation of what a computer sees — it is exactly and only what a computer sees. There is no "line," no "curve," no notion of a digit anywhere in the data. There are twenty-five numbers arranged in a grid. Every technique in this chapter is really answering one question: how do you go from a grid of numbers to the single word "seven," reliably, even when the next photo of a "7" has completely different numbers in it?

Formally, a grayscale digital image is a function I(x, y) that has been sampled on a grid and quantized to integers, producing a matrix of shape H × W (height × width), where each entry is a pixel intensity, usually stored as an 8-bit integer from 0 to 255 rather than the 0–9 scale used above. A colour photograph adds a third dimension: three stacked H × W matrices, one each for the red, green, and blue channels, giving a tensor of shape H × W × 3. A typical phone photo classified by a modern network is first resized to a fixed size — 224 × 224 × 3 is a common convention used by well-known architectures such as ResNet and VGG — which is 150,528 individual numbers before the network has extracted a single feature. Image classification is the task of taking that entire block of numbers and outputting exactly one label from a fixed, known set of classes: not "where is the object" (that is object detection) and not "which pixels belong to it" (that is segmentation) — just one label for the whole image. Keep that boundary sharp; it is one of the most commonly confused ideas in this subject and we will return to it explicitly.

Why Comparing Pixels Directly Fails

The most naive possible classifier is nearest-neighbour matching: store one "template" image per class, and for a new image, measure how different it is from each template using Euclidean (L2) distance, then output the class of the closest template. It is worth actually computing this once so you feel, in numbers, why it doesn't work.

Take a tiny 3×3 crop, using 0 for white and 1 for ink, containing a vertical stroke down the middle column — our template T:

T =  0 1 0        flattened:  T = [0,1,0, 0,1,0, 0,1,0]
     0 1 0
     0 1 0

Now take the exact same stroke, shifted one pixel to the right — call it X. It is the same digit, drawn by the same pen, just not perfectly centred:

X =  0 0 1        flattened:  X = [0,0,1, 0,0,1, 0,0,1]
     0 0 1
     0 0 1

The Euclidean distance between two vectors a and b is d(a,b) = √(Σᵢ(aᵢ − bᵢ)²). Computing it for T and X:

T − X = [0, 1, −1, 0, 1, −1, 0, 1, −1]. Squaring and summing: 0+1+1+0+1+1+0+1+1 = 6. So d(T, X) = √6 ≈ 2.449.

Now compare T against something that isn't the digit at all — a blank white image, Z = [0,0,0,0,0,0,0,0,0]. Here d(T, Z) = √(0²+1²+0²+0²+1²+0²+0²+1²+0²) = √3 ≈ 1.732.

Read that again: √3 < √6. A completely blank image is pixel-wise closer to the template than the exact same stroke shifted by a single pixel. A nearest-neighbour classifier using raw pixel distance would rather call your handwriting "blank" than recognise it as the same "1" you just wrote a hair off-centre. Raw pixel comparison has no concept of translation, rotation, scale, or lighting — a one-pixel shift, a slightly thicker pen, or a shadow across half the page can move an image's pixel vector further from its own class template than from an entirely different class. This is precisely the gap that convolution was designed to close: instead of asking "are these two grids of numbers similar overall," it asks "does this small, local pattern appear somewhere in this image," which survives shifting far better.

Convolution: Extracting Local Features Instead of Comparing Whole Images

A convolutional filter (also called a kernel) is a small matrix — commonly 3×3 or 5×5 — of learnable weights. It is slid across the image, and at every position, the filter's weights are multiplied element-wise against the pixels underneath it and summed to produce one output number. Strictly, this "slide and multiply" operation is cross-correlation; deep learning frameworks call it "convolution" by convention even though true mathematical convolution flips the kernel first. Nothing about the classification result changes if you know which term is technically correct — but knowing the frameworks use the looser name will save you confusion if you read the original literature.

Let's compute one by hand. Take a 5×5 grayscale image with a clean vertical edge — bright on the left, dark on the right:

I =  10 10 10  0  0
     10 10 10  0  0
     10 10 10  0  0
     10 10 10  0  0
     10 10 10  0  0

and a 3×3 vertical-edge kernel K (a Prewitt-style detector):

K = -1  0  1
    -1  0  1
    -1  0  1

Sliding a 3×3 kernel across a 5×5 image produces an output of size (5−3+1) × (5−3+1) = 3×3. Each output cell is the sum of the element-wise product of the kernel with the 3×3 patch beneath it. For the top-left output position, the patch is columns 0–2 of the image (values 10, 10, 10 in every row):

(10)(−1) + (10)(0) + (10)(1) = 0, repeated for all three rows, giving a column sum of 0. For the next patch (columns 1–3, values 10, 10, 0): (10)(−1) + (10)(0) + (0)(1) = −10 per row, giving −30 across three rows. The same holds for columns 2–4. Working through every position gives:

O =   0  -30  -30
      0  -30  -30
      0  -30  -30

Verify this in code — trace it and you'll get exactly the matrix above:

import numpy as np

image = np.array([
    [10,10,10,0,0],
    [10,10,10,0,0],
    [10,10,10,0,0],
    [10,10,10,0,0],
    [10,10,10,0,0]
])
kernel = np.array([[-1,0,1],[-1,0,1],[-1,0,1]])

output = np.zeros((3,3))
for r in range(3):
    for c in range(3):
        patch = image[r:r+3, c:c+3]
        output[r,c] = np.sum(patch * kernel)

print(output)
# [[  0. -30. -30.]
#  [  0. -30. -30.]
#  [  0. -30. -30.]]

Notice what happened: the output is exactly 0 wherever the 3×3 window sits entirely inside the bright region (no edge underneath it), and a strongly negative, constant value wherever the window straddles the boundary between bright and dark — regardless of which row it's in, because the edge runs the same way down every row. The sign is negative here only because this particular kernel responds to "bright on the left, dark on the right"; a kernel with the columns reversed would give +30 instead. In practice this sign issue disappears because the network learns the kernel weights during training rather than us hand-designing them, and it also has an equal opportunity to learn a kernel that fires in the opposite direction. Two things are worth internalising: first, the output is smaller than the input (3×3 from a 5×5 input with a 3×3 kernel — this is why real networks either pad the input or accept that feature maps shrink layer by layer). Second, and more important: this single filter didn't compare the whole image to a template. It detected one specific local pattern — a vertical brightness transition — no matter where in the frame that pattern occurred. That local, position-tolerant detection is exactly what raw pixel-distance matching couldn't do.

The Full Pipeline: From Filters to a Class Probability

A convolutional neural network (CNN) for image classification is a small number of these ideas, stacked and repeated:

• Convolution layer: not one filter but dozens — each layer typically learns 32, 64, or more independent 3×3 kernels, each producing its own feature map. Stacking these feature maps depth-wise is why a single convolution layer's output tensor has shape (height × width × number_of_filters).
• Activation (ReLU): each feature map value x is passed through ReLU(x) = max(0, x), zeroing out negative responses. This injects non-linearity — without it, stacking convolution layers would collapse mathematically into one big linear operation, no more powerful than a single layer.
• Pooling: the feature maps are downsampled, most commonly with 2×2 max pooling — take the maximum of every non-overlapping 2×2 block. Take this 4×4 feature map:

  F =  1  3  2  4
       5  6  1  2
       0  1  8  3
       4  2  3  9

  The four 2×2 blocks (top-left, top-right, bottom-left, bottom-right) have maxima 6, 4, 4, and 9 respectively, giving a pooled 2×2 output of [[6,4],[4,9]]. Pooling does two jobs at once: it shrinks the data (here, by 4×, easing computation in every later layer), and it buys a small amount of translation tolerance — if the strongest edge response shifts by one pixel, it usually still lands in the same 2×2 block and the pooled output doesn't change.
• Repeat: real networks stack several convolution–ReLU–pool blocks. Layer-visualisation research (Zeiler and Fergus, 2014) showed that early layers' filters learn to detect edges and colour blobs — much like our hand-built kernel above — while deeper layers combine those into textures, then object parts (an eye, a wheel), and only the deepest layers respond to whole objects. Nobody hand-designs this hierarchy; it emerges purely from training on labelled examples.
• Flatten and fully connected: the final, small stack of feature maps is unrolled into one long vector x, then passed through a plain matrix multiplication z = Wx + b — exactly the matrix-times-vector operation from your linear algebra syllabus — producing one raw score, or logit, per class. If x has 400 entries and there are 10 classes, W is a 10×400 matrix.
• Softmax: logits are not probabilities — they can be negative or exceed 1. Softmax converts a vector of logits z into a valid probability distribution: softmax(zᵢ) = e^(zᵢ) / Σⱼ e^(zⱼ).

Let's finish the pipeline numerically. Suppose the fully connected layer outputs logits [1.2, 0.3, 4.5] for classes "0," "1," and "7" on our earlier digit:

import math

def softmax(logits):
    exps = [math.exp(z) for z in logits]
    total = sum(exps)
    return [e/total for e in exps]

print(softmax([1.2, 0.3, 4.5]))
# approximately [0.035, 0.014, 0.951]

Trace it: e^1.2 ≈ 3.320, e^0.3 ≈ 1.350, e^4.5 ≈ 90.017. Their sum is ≈94.687, so the three probabilities are 3.320/94.687 ≈ 0.035, 1.350/94.687 ≈ 0.014, and 90.017/94.687 ≈ 0.951 — they sum to 1.000 as any probability distribution must. The network reports "7" with 95.1% confidence. This entire journey — a grid of pixel intensities, through learned filters, through pooling, through one matrix multiplication, through softmax — is what "image classification" means end to end.

How the Filters Learn: Loss, Training Data, and Why Augmentation Has Rules

Nothing about the kernel values in the worked example above was hand-picked by an engineer in a real system — they start as small random numbers and are adjusted by gradient descent to minimise a loss function computed over thousands of labelled examples. For classification, the standard loss is cross-entropy: L = −ln(p_{true class}), where p_{true class} is the softmax probability the network assigned to the correct label. If the true label for our digit was indeed "7" and the network assigned it 0.951, the loss is −ln(0.951) ≈ 0.050 — small, because the network was right and confident. If instead the network had assigned "7" only 0.10 probability, the loss would be −ln(0.10) ≈ 2.303 — a much larger penalty, and the gradient of that loss with respect to every filter weight (computed via backpropagation, the chain rule applied layer by layer) tells each weight how to nudge itself to reduce the error next time. Millions of these small nudges, repeated over many passes through the training data, are what turn random noise into edge detectors.

This only works if the network never gets to "memorise the answer key." Datasets are split into a training set (used for the gradient updates), a validation set (used to tune settings like how many filters or how long to train, without touching the actual test data), and a held-out test set (touched exactly once, to report the final honest accuracy). A network that scores 99% on training images but 60% on the test set has overfit — it has memorised its 60,000 specific training photographs rather than learning the general concept "sevenness." A standard countermeasure specific to computer vision is data augmentation: synthetically multiplying the training set by randomly flipping, rotating a few degrees, cropping, or adjusting the brightness of each image before it's shown to the network, forcing the filters to become tolerant to exactly the kinds of variation that broke naive pixel-distance matching earlier in this chapter. But augmentation must respect what the label actually means: flipping a photo of a cat horizontally is harmless — it's still obviously a cat — but flipping a training image of the digit "2" or a "7" horizontally produces a shape that either isn't a valid digit at all or reads as a different one, and training on it would actively teach the network something false. Good augmentation choices are always specific to the task, not a checklist applied blindly.

Two Misconceptions Worth Correcting Explicitly

Misconception 1 — "Image classification finds where the object is." It does not. Classification outputs exactly one label for the entire image: "this photo contains a dog," full stop, with no location. If you need a box around the dog, that's object detection (models like YOLO or Faster R-CNN, which additionally output bounding-box coordinates and can find multiple objects per image). If you need to know exactly which pixels belong to the dog, that's semantic segmentation. All three tasks reuse the same convolutional feature-extraction backbone described in this chapter, which is exactly why students conflate them — but classification's output layer is a single softmax over a fixed class list, nothing more. On CBSE and competitive papers, a question describing bounding boxes or per-pixel masks is not testing image classification, even if it's dressed up as one.

Misconception 2 — "The network understands what a cat is." A trained classifier has found statistical correlations between certain pixel patterns and a label; it has not acquired any concept of "cat" the way you have one. The clearest proof is the existence of adversarial examples: Szegedy et al. (2013) showed that adding a carefully computed pattern of noise to a correctly classified image — a perturbation so small a human cannot even perceive it — can make a state-of-the-art classifier confidently output a completely wrong label. No cat "changed" in that image in any way a human would notice; only its pixel statistics shifted in a direction the network's decision boundary happens to be sensitive to. If the network truly understood "catness" as a concept, an imperceptible nudge to individual pixel values could not flip its answer. This is not a minor technical footnote — it is a live, actively researched vulnerability in real deployed vision systems, and it is the single best piece of evidence that "the network sees like we do" is false.

A Landmark Result Worth Knowing: AlexNet, 2012

Every year from 2010, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) tested classifiers on roughly 1.2 million training images spanning 1,000 categories. Through 2011, the best systems used hand-engineered features (like SIFT descriptors) feeding into classical classifiers, improving gradually. In 2012, a deep CNN called AlexNet — designed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto — entered the competition and won by a margin that stunned the field: a top-5 error rate of roughly 15%, against the next-best (non-neural) entry's roughly 26%. A 10-percentage-point jump in a single year, on a benchmark that had been improving by fractions of a percent, is widely regarded as the moment that convinced the broader research community deep convolutional networks — the exact convolution-pool-flatten-softmax pipeline built by hand in this chapter — were not a niche technique but the future of computer vision. Every major vision architecture since (VGG, ResNet, EfficientNet, and the vision transformers used alongside CNNs today) is answering the same convolution-based question this chapter walked through, just with far more layers and far more data.

Where This Fits: CBSE and Competitive Exams

Computer Vision is one of the core technology domains in CBSE's Artificial Intelligence curriculum (skill subject 417 at the secondary level), alongside Data Science and Natural Language Processing — expect board questions on the CV pipeline stages (input, feature extraction, classification), on distinguishing classification from detection and segmentation, and on real applications (quality inspection, medical imaging, agricultural crop monitoring). For competitive exams: JEE and BITSAT do not test CNN architectures directly, but every convolution you computed by hand above is a matrix operation — element-wise multiplication and summation over a window — and the fully connected layer is literally Wx + b, the matrix-vector product you drill for JEE's matrices and determinants unit. Treat this chapter's arithmetic as disguised linear algebra practice, not a separate subject. At the GATE-foundation level, convolution and pooling are core topics under Digital Image Processing and Machine Learning papers, formalised with the same definitions used here. For Olympiad-style informatics preparation, the transferable skill isn't CNN trivia — it's comfort translating a sliding-window computation (like the 3×3-over-5×5 convolution above) into correct nested-loop code, which is exactly what the NumPy snippet in this chapter does.

Check Your Understanding

1. Why does Euclidean pixel-distance nearest-neighbour matching call a one-pixel-shifted "1" less similar to a template "1" than a blank white image is? Because L2 distance penalises every mismatched pixel equally regardless of cause; a shift moves ink into pixels the template has as background and vice versa, creating many mismatches, while a blank image merely lacks the ink pixels — coincidentally fewer squared differences in this small example. It has no notion that "shifted ink" and "the same stroke" are related.
2. A 4×4 feature map is [[2,1,0,3],[4,7,2,1],[5,0,1,6],[1,2,8,3]]. Compute the result of 2×2 max pooling with stride 2. Top-left block {2,1,4,7} → 7. Top-right {0,3,2,1} → 3. Bottom-left {5,0,1,2} → 5. Bottom-right {1,6,8,3} → 8. Pooled output: [[7,3],[5,8]].
3. A model outputs logits [0.5, 3.1, 0.2] for classes "cat," "dog," "fox." Find the predicted class and its confidence. e^0.5≈1.649, e^3.1≈22.198, e^0.2≈1.221; sum≈25.068. Probabilities ≈ [0.066, 0.886, 0.049]. Predicted class: "dog," at about 88.6% confidence.
4. A vision system draws a rectangle around every pedestrian in a street photo. Is this image classification? Justify your answer. No — classification outputs one label for the whole image with no location information. Drawing per-object boxes is object detection, a related but distinct task built on the same convolutional backbone.
5. Why is horizontal-flip augmentation safe for a "cat vs. dog" dataset but unsafe for a handwritten-digit dataset that includes "2" and "7"? A mirrored cat is still recognisably a cat, so the label stays correct. A mirrored "2" or "7" is no longer a valid instance of that digit (and may resemble a different symbol entirely), so flipping would train the network on images paired with a now-false label.

Summary

An image, to a computer, is nothing but a matrix (or a stack of three, for colour) of pixel intensities — there is no built-in notion of shape, edge, or object. Comparing these matrices directly with a distance metric like Euclidean distance fails because it has no tolerance for shifting, rotation, or lighting changes, as the "blank image beats a shifted digit" calculation demonstrated concretely. Convolutional filters solve this by detecting small, local patterns — edges, then textures, then object parts, then whole objects — wherever they occur in the frame, with the exact arithmetic (element-wise multiply, sum, slide) shown in the worked 5×5 example. Max pooling shrinks these feature maps while adding a further layer of positional tolerance. A final flatten-and-fully-connected step reduces everything to one score per class, and softmax turns those scores into a genuine probability distribution. The whole stack is trained by minimising cross-entropy loss via gradient descent and backpropagation, evaluated honestly only on data it never trained on, and made more robust through label-respecting data augmentation. None of this amounts to understanding in the human sense — adversarial examples prove the network is tracking pixel statistics, not concepts — but the 2012 AlexNet result showed that this statistical, layered, learned-filter approach outperforms hand-engineered computer vision by a wide enough margin to have reshaped the entire field.

Think About It

Think about this: How would you explain image classification: teaching machines to see to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.