Convolutional Neural Networks: How Your Phone Recognizes Your Face

The 200-Millisecond Problem

Pick up a phone with face unlock, look at it, and the screen opens before you have consciously finished the act of looking. That gap is a few hundred milliseconds. In that time, the camera has captured an image, a program has decided "this is the registered owner's face, not a photo, not a stranger, not a sibling who looks similar," and the phone has acted on that decision. There is no human anywhere in that loop. Somewhere inside the phone's processor, a piece of arithmetic looked at a grid of brightness values and produced a yes.

This chapter is about exactly what that arithmetic is. It is called a Convolutional Neural Network, or CNN, and despite the intimidating name, the core operation it repeats thousands of times is something you can compute by hand with multiplication and addition. By the end of this chapter you will be able to compute one step of it yourself, trace a real (short) piece of code that performs it, and explain precisely why this particular trick — and not an ordinary neural network — is what makes fast, reliable image understanding possible on a device that fits in your pocket.

Step One: An Image Is Just a Grid of Numbers

Before any network can "look" at a picture, the picture has to become numbers. A digital photo is stored as a grid of tiny squares called pixels. In a grayscale image, each pixel holds one number describing how bright it is — commonly on a scale from 0 (pure black) to 255 (pure white), though for the arithmetic in this chapter we will use a simplified 0-to-1 scale, where 0 means dark and 1 means bright. A colour photo just uses three such grids stacked together, one each for red, green and blue intensity — but everything you learn here about a single grayscale grid extends directly to that case, so we will stick with grayscale to keep the arithmetic clean.

So a 5-pixel-by-5-pixel patch of a photograph — five rows, five columns — is really nothing more than a 5×5 table of numbers. Here is one, representing a small patch where the left two columns are bright and the right three columns are dark, the kind of pattern you would see at the boundary of a bright forehead against a darker background:

1  1  0  0  0
1  1  0  0  0
1  1  0  0  0
1  1  0  0  0
1  1  0  0  0

A real face-unlock photo is nowhere near this small — it might be a hundred pixels or more on each side after the phone crops in on your face — but the principle for a 5×5 patch and a 100×100 photo is identical. Once you can see an image as "just a table of numbers," the question of how a computer "recognizes a face" stops being mysterious and becomes a specific, answerable question: what arithmetic turns a table of brightness numbers into the decision "this is a face" and then "this is the correct face"?

Why an Ordinary Neural Network Struggles Here

If you have already met the basic neural network — layers of neurons, each connected to every neuron in the layer before it, each connection carrying a weight — the obvious first idea is to feed the image's pixel numbers straight in as inputs, one input per pixel. This does not work well for images, and the reason is worth working out with real numbers, not just taking on faith.

Suppose the phone crops your face to a modest 100×100 grayscale image. That is 100 × 100 = 10,000 pixels, so a fully connected first layer needs 10,000 inputs. If that first hidden layer has even a modest 100 neurons, and every one of those neurons connects to every one of the 10,000 inputs, the layer needs 10,000 × 100 = 1,000,000 weights — a million numbers the network has to learn correctly, just to get through the first layer of one image. A phone chip re-running this every time you glance at the screen, many times a day, cannot afford that.

The parameter count is only half the problem. The deeper issue is that a fully connected layer treats pixel number 347 and pixel number 8,219 as no more related to each other than any other pair of numbers — it has no built-in notion that pixels sitting next to each other in the grid describe the same physical patch of your face and should be examined together. It also has no way to reuse what it learns: if it learns to detect an eyebrow using the specific set of weights attached to pixels near the top-left of the frame, and then your face shifts slightly right in the next frame, those exact pixels no longer contain an eyebrow, and the network has to have separately learned an entirely different set of a million-plus weights to catch an eyebrow appearing there instead. Nothing is shared. Every position in the image is, to a fully connected layer, unrelated territory.

What we actually want is a small pattern-detector — "does this patch of pixels look like an eyebrow edge?" — that can be swept across the entire image and applied identically everywhere, learning the pattern once and reusing it in every position. That single idea, sweeping a small detector across a grid and reusing its weights everywhere, is the entire core of a CNN. It is called convolution.

The Convolution Operation, Worked by Hand

A convolution filter (also called a kernel) is a small grid of numbers — commonly 3×3 — that gets placed on top of a patch of the image, multiplied position-by-position with the pixels underneath it, and the nine products are added into a single number. That number becomes one entry of the output. Then the filter slides one column over and does it again, and again, sweeping across the whole image.

Let's use a real filter and compute real numbers. Take this 3×3 filter, which is built to detect vertical edges — places where brightness changes from left to right across a small patch:

 1   0  -1
 1   0  -1
 1   0  -1

Notice the shape: it rewards a patch whose left column is bright and whose right column is dark (a positive number), it is indifferent to the middle column (multiplied by 0), and it produces a negative number for the opposite pattern. It is, in effect, a tiny hard-coded question: "is the left side of this patch brighter than the right side?"

Now place this filter over the top-left 3×3 corner of our 5×5 image from before:

Image patch:      Filter:
1  1  0            1   0  -1
1  1  0            1   0  -1
1  1  0            1   0  -1

Multiply each pair of numbers in matching positions, then add all nine products:

(1×1)+(1×0)+(0×-1)
+ (1×1)+(1×0)+(0×-1)
+ (1×1)+(1×0)+(0×-1)
= 1 + 1 + 1 = 3

That single number, 3, is the first entry of the output. Now slide the filter one column to the right, so it covers columns 1–3 instead of 0–2, and repeat: the patch is now [1,0,0] in every row, giving (1×1)+(0×0)+(0×-1) = 1 per row, summed over three rows = 3 again. Slide once more to cover columns 2–4, where every pixel underneath is 0, and the sum is 0.

Because our 5×5 image is only 5 pixels wide and the filter is 3 pixels wide, it can only occupy 3 distinct horizontal positions (starting at column 0, 1, or 2) before it runs off the edge — the general rule is output width = image width − filter width + 1, here 5 − 3 + 1 = 3. The same rule applies vertically, so sliding the filter fully across all rows and columns of this particular image (which happens to look identical in every row) produces this 3×3 output grid, called a feature map:

3  3  0
3  3  0
3  3  0

Read this output correctly: a large number means "the filter's pattern — bright-left, dark-right — is strongly present here." A number near zero means "this patch was fairly uniform, no left-right brightness change." Notice the edge produces a band of large values (both output columns 0 and 1 read 3), not a single sharp spike — that is expected and correct, not an error: because the filter is 3 pixels wide, any window that straddles the boundary between the bright and dark regions detects it, and two consecutive window positions both straddle a boundary that is only one pixel wide.

Checking the Arithmetic in Code

Here is the entire convolution operation written as plain Python, with no machine-learning library — just nested loops doing exactly the multiplication-and-addition we did by hand above.

def convolve2d(image, kernel):
    img_h, img_w = len(image), len(image[0])
    k_h, k_w = len(kernel), len(kernel[0])
    out_h = img_h - k_h + 1
    out_w = img_w - k_w + 1
    output = [[0] * out_w for _ in range(out_h)]
    for i in range(out_h):
        for j in range(out_w):
            total = 0
            for di in range(k_h):
                for dj in range(k_w):
                    total += image[i + di][j + dj] * kernel[di][dj]
            output[i][j] = total
    return output

image = [
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 0],
]
kernel = [
    [1, 0, -1],
    [1, 0, -1],
    [1, 0, -1],
]
print(convolve2d(image, kernel))

Trace it for output position i=0, j=0: the inner loops walk di and dj from 0 to 2, reading image[0][0]…image[2][2] against kernel[0][0]…kernel[2][2]. That is exactly the 3×3 patch and filter we multiplied by hand, and total accumulates to 3, matching output[0][0] = 3. Because every row of our test image is identical, every row of the printed result is identical too, so running this prints [[3, 3, 0], [3, 3, 0], [3, 3, 0]] — precisely the feature map we computed on paper. This is the whole trick: a CNN's "convolutional layer" is this same double loop, run with a filter whose nine numbers were learned from thousands of training photographs rather than chosen by hand.

Seeing the Slide

What a Filter Actually Detects, and Why We Need Many of Them

The filter we just used detects one specific pattern: a vertical boundary between a bright region on the left and a dark region on the right. That is genuinely useful — a face has plenty of such boundaries, at the edge of the nose against the cheek, at the hairline, along the jaw — but it is only one pattern out of the hundreds that matter for recognizing a face. A real convolutional layer does not apply a single filter; it applies many filters in parallel — commonly 32 or 64 in an early layer — each with its own independently learned set of 9 numbers, one detecting horizontal edges, another detecting diagonal edges, another responding to a particular curve, another to a patch of a certain colour or texture. Each filter produces its own feature map, so a layer with 32 filters turns one input image into 32 separate feature maps, each highlighting where a different micro-pattern occurs.

This is also where the parameter-count advantage becomes concrete. Our filter had 9 weights (plus, in practice, one additional "bias" number added to every output, so 10 parameters total). A convolutional layer with 32 such filters uses 32 × 10 = 320 parameters, no matter whether the input image is 5×5 or 500×500 — compare that to the 1,000,000 weights the fully connected layer needed for a single 100×100 image. That is over 3,000 times fewer numbers to learn and store, while additionally gaining the property that a pattern learned in one part of the image is automatically recognized anywhere else it appears, because the exact same 9 numbers slide across every position.

ReLU: Keep the Evidence, Discard the Contradiction

After a convolutional layer produces its feature maps, each number is passed through a very simple function called ReLU (Rectified Linear Unit), defined as ReLU(x) = max(0, x) — if the number is positive, leave it unchanged; if it is negative, replace it with zero.

Why bother? Recall what a negative output from our edge filter means: the filter is built to report "bright-left, dark-right," so a positive number means that pattern is present, while a negative number means the opposite pattern (dark-left, bright-right) is present. If we slid our filter over a patch where the pattern were reversed — say a patch of [0, 0, 1] in every row instead of [1, 1, 0] — each row would contribute (0×1) + (0×0) + (1×-1) = -1, giving a total of -3. That -3 is a real, meaningful number, but it is not evidence for the specific thing this filter exists to detect; it is evidence for a different pattern that some other filter is responsible for. Passing -3 through ReLU gives max(0, -3) = 0, so this filter's feature map simply reports "nothing here" rather than a misleading negative signal. Without a step like this, positive and negative evidence from many filters could cancel each other out when combined in later layers, blurring signals that should stay separate. ReLU is also what makes a CNN capable of learning anything beyond a simple straight-line relationship between pixels and the final answer — stacking layers of pure multiplication and addition without it would mathematically collapse back into one single linear operation, no matter how many layers you stacked.

Pooling: Shrinking the Map Without Losing the Point

Feature maps from an early convolutional layer are almost as large as the original image, and a real network stacks many convolutional layers, so the amount of data would balloon if nothing were done to control it. Pooling solves this by shrinking each feature map, most commonly with max pooling: slide a small window — typically 2×2 — across the feature map and keep only the single largest value in each window, discarding the rest.

Take this 4×4 feature map (imagine it is the output of some filter after ReLU, where larger numbers mean stronger detection of whatever that filter looks for):

3  1  0  2
5  4  1  0
0  1  8  2
3  2  1  6

Split it into four non-overlapping 2×2 blocks and take the maximum of each: the top-left block is {3, 1, 5, 4}, maximum 5; the top-right block is {0, 2, 1, 0}, maximum 2; the bottom-left block is {0, 1, 3, 2}, maximum 3; the bottom-right block is {8, 2, 1, 6}, maximum 8. The pooled output is:

5  2
3  8

Here is the same operation as code, which you can trace exactly the way we traced the convolution:

def max_pool2x2(fmap):
    h, w = len(fmap), len(fmap[0])
    out = []
    for i in range(0, h, 2):
        row = []
        for j in range(0, w, 2):
            block = [fmap[i][j], fmap[i][j + 1],
                     fmap[i + 1][j], fmap[i + 1][j + 1]]
            row.append(max(block))
        out.append(row)
    return out

fmap = [
    [3, 1, 0, 2],
    [5, 4, 1, 0],
    [0, 1, 8, 2],
    [3, 2, 1, 6],
]
print(max_pool2x2(fmap))

Tracing it: when i=0, j=0, block reads fmap[0][0]=3, fmap[0][1]=1, fmap[1][0]=5, fmap[1][1]=4, and max(block) is 5. When i=2, j=2, block reads fmap[2][2]=8, fmap[2][3]=2, fmap[3][2]=1, fmap[3][3]=6, giving 8. The full result prints as [[5, 2], [3, 8]], matching our hand computation exactly. A 4×4 map became a 2×2 map — a quarter of the data — while keeping the strongest activation from each region.

Pooling has a second benefit beyond compression: it makes the network mildly tolerant of small shifts. If your face is one pixel further left in one photo than another, the strongest activation for "eyebrow edge" might land in a slightly different exact position, but as long as it still falls within the same 2×2 pooling window, the pooled output is identical. This is part of why face unlock keeps working even though you never hold the phone at exactly the same angle twice.

Correcting a Common Misconception

A very natural but incorrect mental model is: "the phone has a stored photo of my face, and it compares the new camera image to that photo pixel by pixel, like laying two transparencies on top of each other." If this were true, face unlock would fail constantly — the lighting is never identical twice, your head is never at the exact same angle, glasses come on and off, and a direct pixel-by-pixel comparison would register all of that as a mismatch.

What actually happens is closer to the opposite. The stack of convolutional and pooling layers we just built, repeated several times with increasing numbers of filters, gradually compresses the image down to a short list of numbers — often a few hundred — called a face embedding or feature vector. These numbers are not pixels; they are the network's learned summary of higher-level properties (something closer to "how far apart are the eyes relative to nose width," expressed in a way only the network's own internal numbers can interpret directly). The phone compares this new embedding to the embedding it stored when you enrolled your face, using a mathematical distance between the two lists of numbers, and unlocks if that distance is smaller than a set threshold. Because the network was trained on huge numbers of photos of the same people under different lighting and angles, it has learned to produce embeddings that stay close together for the same person despite those changes — robustness to lighting and pose is a property the network was trained to have, not a side effect of storing more photos. This is also precisely why pooling's "small shift tolerance" matters at every layer on the way to that final embedding, not just once at the end.

A second, smaller misconception worth correcting directly: pooling is not the same as blurring or averaging. Average pooling exists and is used in some designs, but the max pooling we computed above deliberately keeps only the single strongest signal in each window and throws the rest away — it is closer to "did this small region contain strong evidence of the pattern, yes or no" than to smoothing the image.

Stacking Layers: From Edges to a Face

A single convolutional layer only sees 3×3 patches, far too small to represent an eye, let alone a face. The power of a CNN comes from stacking many convolution-ReLU-pooling blocks one after another. The first layer's filters, working directly on pixels, learn simple things — short edges and blobs of colour, much like our worked example. The second layer's filters do not see raw pixels at all; they see the output of the first layer, meaning they operate on combinations of edges, and so they can learn to respond to a corner, a curve, or a short line segment made of several first-layer edges lined up together. A third layer, seeing combinations of those, can respond to something recognisable as an eye corner, a nostril, or the curve of a jaw. By the time you reach the final convolutional layers, filters are responding to arrangements of parts that correspond to a specific face's structure.

From Feature Maps to a Decision: Flatten, Dense, Output

After the last pooling layer, we are left with a stack of small feature maps rather than a single answer. The next step is called flattening: every number in every remaining feature map is laid out into one long list. That list is then fed into one or more ordinary fully connected ("dense") layers — the kind of layer that would have been hopeless applied directly to raw pixels, but is now applied only to a few hundred already-distilled numbers, so the parameter-count problem from earlier in the chapter no longer applies. The final dense layer produces the embedding vector, or in a simple classification CNN, a set of output numbers, one per possible category, indicating how strongly the network believes the image belongs to each category.

Face unlock is a slightly different setup than the classic "classify this photo as a cat or a dog" example, and it is worth being precise about why: your phone was never trained to output "the owner" versus "everyone else on Earth" as fixed categories, because it has never seen photos of everyone else on Earth, and it cannot be retrained every time a new person buys the same phone model. Instead, the CNN is trained once, on a very large and varied dataset of faces, purely to become good at producing embeddings that place photos of the same person close together and photos of different people far apart, without caring who any of them are. Your specific phone then simply records your embedding once during setup and, every time you look at it, computes a fresh embedding and measures the distance to the one on file. Many phones combine this with additional hardware — an infrared camera or a depth sensor projecting a dot pattern onto your face — precisely because a single 2-D CNN judgement, however good, can in principle be fooled by a flat photograph; depth data gives the system independent evidence that it is looking at a real three-dimensional face, not a picture of one.

Where This Shows Up in India

Two real, currently operating Indian deployments use exactly this pipeline. Digi Yatra, run under the Ministry of Civil Aviation's framework and launched in December 2022, lets a registered passenger walk through airport entry and boarding checkpoints at major airports by having a camera capture their face and match it against the identity linked at registration, instead of a staff member checking a paper boarding pass and ID at every gate — the face-matching step is a CNN-based embedding comparison of exactly the kind described above. Separately, the UIDAI's Aadhaar system has, since 2018, supported face authentication as an additional way to verify identity for services such as e-KYC, alongside fingerprint and iris scanning, specifically to help people whose fingerprints are hard to read (due to age or manual labour) still be able to authenticate. In both cases, the system is not storing and re-displaying your photo for a human to check — it is running a CNN to produce an embedding and comparing that embedding mathematically, the same computation you traced by hand and in code earlier in this chapter.

Check Your Understanding

1. (MCQ) Why is a convolutional layer able to use far fewer parameters than a fully connected layer examining the same image?
   (a) Because it only looks at a smaller image
   (b) Because the same small filter's weights are reused at every position instead of having separate weights for each pixel
   (c) Because it skips most of the pixels
   (d) Because it does not use multiplication
   (Answer: b — reuse of the same 9-or-so weights across every sliding position is exactly what keeps the parameter count independent of image size.)
2. (2 marks) In your own words, explain why passing a convolution output through ReLU, rather than passing it through unchanged, is useful when several filters' feature maps are later combined.
3. (Numerical, work it out before checking) Apply the filter [[1,0,-1],[1,0,-1],[1,0,-1]] to this 3×3 image patch and find the single output number:

   2  4  6
   1  3  5
   0  2  4

   Worked solution: row 1 gives (2×1)+(4×0)+(6×-1) = -4; row 2 gives (1×1)+(3×0)+(5×-1) = -4; row 3 gives (0×1)+(2×0)+(4×-1) = -4. Total = -12. The negative sign makes sense: in every row, the right-hand pixel is brighter than the left-hand one — the opposite of the bright-left pattern this filter was built to reward.
4. (True/False, with correction) "Max pooling keeps the average brightness of each window." State whether this is true or false, and correct it if false.
   (Answer: False — max pooling keeps only the single largest value in each window; it discards the rest rather than averaging them.)
5. (Applied, 3–4 sentences) Digi Yatra uses a CNN-based pipeline like the one in this chapter to match a traveller's face at an airport gate. Suppose the pooling layers were removed from such a system, leaving only convolution and ReLU layers stacked directly on top of each other. Explain what would happen to (a) the amount of computation the airport's system needs to perform per traveller, and (b) how sensitive the match would be to the traveller's head being tilted by a few degrees compared to their enrollment photo.

Summary

• An image is a grid of brightness numbers; a face-unlock decision is ultimately arithmetic performed on that grid, not literal "vision."
• A fully connected network applied directly to image pixels needs an impractical number of weights (a million-plus for a modest 100×100 image) and cannot recognize a pattern that shifts position without relearning it from scratch.
• Convolution fixes this by sliding one small filter (e.g., 3×3, 9-10 numbers) across the whole image, computing a dot product at each position, and reusing the identical weights everywhere — producing a feature map that highlights where that filter's specific pattern occurs.
• ReLU (max(0, x)) discards negative "opposite pattern" evidence after each convolution, and is also what lets stacked layers learn more than a straight-line relationship.
• Max pooling shrinks feature maps by keeping only the strongest value in each small window, cutting computation and adding tolerance to small shifts — it is not an average or a blur.
• Stacking convolution-ReLU-pooling blocks builds a hierarchy: early layers detect edges, middle layers detect curves and corners built from those edges, later layers detect face parts built from those curves.
• The final layers flatten the last feature maps into a list of numbers and pass them through dense layers to produce a compact face embedding; face unlock compares this embedding's distance to a stored one, rather than comparing photos pixel by pixel — which is why it survives changes in lighting, angle, and expression.
• Digi Yatra (Indian airports, since December 2022) and Aadhaar face authentication (UIDAI, since 2018) are real, currently running Indian systems built on exactly this convolution-pooling-embedding pipeline.