Image Processing: Pixels and Filters

Every year, lakhs of students across India fill OMR sheets — those bubble sheets used in exams like NTSE, JEE Main, and NEET. A machine scans the sheet and, in a fraction of a second, decides which bubble you filled. It has no eyes and no understanding of what a "pencil mark" is. All it has is a grid of numbers. Understanding how a machine turns a photograph into numbers — and then does useful things with those numbers — is the whole subject of image processing. By the end of this chapter, you will be able to write the actual arithmetic a scanner performs to read a bubble, and the arithmetic a photo-editing app performs to blur or sharpen a picture.

What Is a Pixel, Really?

Open any photo on your phone and zoom in again and again. At some point, the smooth image breaks apart into a mosaic of tiny solid-colored squares. Each of those squares is a pixel (short for "picture element"). A digital image is not a continuous picture the way a painting is — it is a rectangular grid of pixels, and every pixel stores a small set of numbers describing its color.

The simplest case is a grayscale (black-and-white) image. Each pixel stores a single number between 0 and 255:

• 0 means pure black.
• 255 means pure white.
• Numbers in between are shades of gray — the closer to 0, the darker; the closer to 255, the lighter.

Why 0 to 255, and not some other range? Because computers store each grayscale value in one byte of memory, and one byte has exactly 2^8 = 256 possible values, numbered 0 through 255. This is not an arbitrary design choice — it is a direct consequence of how memory is organized in binary, and you will see the number 255 constantly in image processing for this reason.

An image, then, is nothing but a grid (a list of lists, if you are thinking like a programmer) of these 0–255 numbers. A photo that is 1920 pixels wide and 1080 pixels tall is a grid with 1920×1080 = 2,073,600 numbers in it. "Processing an image" simply means running arithmetic over this grid of numbers.

A Worked Example: Reading an OMR Bubble

Let's make this concrete with the OMR scanner. When the scanner photographs a small square patch of the answer sheet where one bubble is expected, it gets a tiny grid of grayscale numbers. Suppose the patch is only 4×4 pixels for simplicity (a real scanner would use more, but the idea is identical).

Here is the patch over a bubble that was not filled in — just blank white paper with faint scanner noise:

250 245 248 252
255 240 250 245
248 252 255 250
245 250 248 255

And here is a patch over a bubble that was filled in with a dark HB pencil:

30 40 35 45
25 20 30 40
50 45 35 25
40 30 45 50

How does the scanner decide, automatically, which is which? The simplest possible rule — and a genuine one used in real OMR systems — is to compute the average intensity of the patch and compare it to a fixed cutoff, called a threshold. This single idea, "compute one number from the pixels, then compare it to a cutoff," is the simplest kind of image filter there is.

Let's compute the averages by hand first, so the code that follows is not a black box.

For the unfilled patch: add all 16 numbers. Row by row: 250+245+248+252 = 995; 255+240+250+245 = 990; 248+252+255+250 = 1005; 245+250+248+255 = 998. Total = 995+990+1005+998 = 3988. Average = 3988 ÷ 16 = 249.25.

For the filled patch: 30+40+35+45 = 150; 25+20+30+40 = 115; 50+45+35+25 = 155; 40+30+45+50 = 165. Total = 150+115+155+165 = 585. Average = 585 ÷ 16 = 36.5625.

If we set the threshold at 128 (the exact midpoint of the 0–255 range), the rule is: "average below 128 means the bubble is filled; average at or above 128 means it is empty." 249.25 ≥ 128, so the first patch is correctly read as empty. 36.5625 is well below 128, so the second is correctly read as filled. This is exactly the kind of decision an OMR scanner makes for every single bubble on every sheet, millions of times a year.

Here is that same logic as working Python code, matching the numbers above exactly:

def average_intensity(patch):
    total = 0
    count = 0
    for row in patch:
        for pixel in row:
            total += pixel
            count += 1
    return total / count

def is_filled(patch, threshold=128):
    return average_intensity(patch) < threshold

unfilled_patch = [
    [250, 245, 248, 252],
    [255, 240, 250, 245],
    [248, 252, 255, 250],
    [245, 250, 248, 255]
]

filled_patch = [
    [30, 40, 35, 45],
    [25, 20, 30, 40],
    [50, 45, 35, 25],
    [40, 30, 45, 50]
]

print(average_intensity(unfilled_patch))  # 249.25
print(is_filled(unfilled_patch))          # False
print(average_intensity(filled_patch))    # 36.5625
print(is_filled(filled_patch))            # True

Trace it yourself: average_intensity walks every row, then every pixel in that row, adding to total and incrementing count. After the loops finish, total is 3988 and count is 16 for the first patch, so it returns 3988/16 = 249.25 — exactly matching the hand calculation above.

Color Images: Three Grids Stacked Together

A color pixel is not one number — it is three: how much Red, how much Green, and how much Blue, each again on the 0–255 scale. This is the RGB model, and it works because human eyes have three types of color receptors sensitive to roughly red, green, and blue light; mixing these three in different proportions reproduces almost any color we can perceive.

Take the saffron band of the Indian flag, which uses the standard color code #FF9933 — in decimal, that's R=255, G=153, B=51. A color image is really three grids stacked on top of each other: an R-grid, a G-grid, and a B-grid, all the same width and height, describing the same photo.

To convert a color pixel to grayscale (which is what many filters, including our OMR thresholding, need to do first), you cannot just average the three numbers — human eyes are far more sensitive to green light than to red or blue. The standard formula, used in broadcast television and most image libraries, weights the channels unevenly:

gray = 0.299 * R + 0.587 * G + 0.114 * B

For the saffron pixel (255, 153, 51): 0.299×255 = 76.245; 0.587×153 = 89.811; 0.114×51 = 5.814. Sum = 76.245 + 89.811 + 5.814 = 171.87, which rounds to 172 — a fairly light gray, matching how bright saffron looks. Compare this to the flag's green band, #138808 (R=19, G=136, B=8): 0.299×19 = 5.681; 0.587×136 = 79.832; 0.114×8 = 0.912. Sum = 86.425, which rounds to 86 — noticeably darker, matching how the green band looks duller than the saffron under the same lighting. Notice how the green channel dominates both results because of its 0.587 weight — this is why the formula is not a simple average.

Two Families of Filters: Point Filters and Neighborhood Filters

An image "filter" is any rule that takes the numbers in an image and produces new numbers. Once you accept that an image is just a grid of numbers, "applying a filter" stops being mysterious — it becomes ordinary arithmetic on that grid. There are two fundamentally different families of filters, and separating them clearly is the single most useful idea in this chapter.

Point filters compute a new value for each pixel using only that pixel's own value — never looking at its neighbors. Thresholding a single pixel, or brightening every pixel by adding a fixed amount, are point filters.

Neighborhood filters compute a new value for each pixel using that pixel and the pixels around it. Blurring and edge detection are neighborhood filters — you cannot blur a single isolated pixel, because blurring means "mix a pixel with its surroundings," and a pixel with no surroundings has nothing to mix with.

Point Filters: One Pixel at a Time, and the Overflow Trap

Suppose you want to brighten a photo by adding 20 to every pixel's grayscale value. Simple enough — except what happens when a pixel is already 240? 240 + 20 = 260, but a grayscale value can only be 0–255. There is no such thing as "260" in an 8-bit image. This is not a rare edge case; bright skies and white paper in real photos are full of pixels near 255, so this problem comes up constantly.

Here is where a genuine, well-documented bug lives in real image-processing code. Many low-level image libraries store pixels as an 8-bit unsigned integer type. If you add without checking the range, the value doesn't become 260 — the extra bit is silently dropped and the number wraps around: 260 becomes 260 − 256 = 4. A near-white pixel (240) that should have become brighter instead becomes almost pure black (4) — a jarring, visible glitch, and a classic real bug in unsafe image code.

def add_brightness_unsafe(value, amount):
    return (value + amount) % 256   # mimics 8-bit wraparound

def add_brightness_safe(value, amount):
    result = value + amount
    if result > 255:
        return 255
    if result < 0:
        return 0
    return result

print(add_brightness_unsafe(240, 20))  # 4   -- wrapped around, a bug!
print(add_brightness_safe(240, 20))    # 255 -- correctly clipped
print(add_brightness_safe(10, -30))    # 0   -- correctly clipped at the dark end

Trace add_brightness_unsafe(240, 20): 240 + 20 = 260, and 260 % 256 = 4, because 256 goes into 260 once with remainder 4. That is exactly the wraparound bug described above. Now trace add_brightness_safe(240, 20): result is 260, which is greater than 255, so the function returns 255 — the value is "clipped" or "clamped" to the valid range instead of overflowing. This clip-to-range step, using max(0, min(255, value)) or the equivalent if-statements shown here, is mandatory in every correct point filter that adds or multiplies pixel values.

Neighborhood Filters: Convolution and Blurring

Now consider blurring — smoothing out sensor noise or small dust specks in a scanned image. The standard tool is a box blur: replace each pixel with the average of itself and its 8 immediate neighbors (a 3×3 block of 9 pixels total). This process of sliding a small block of weights across an image and combining it with the pixels underneath is called convolution, and the small block of weights is called a kernel. For a box blur, the kernel is simply nine equal weights of 1/9 each — meaning "add up these 9 pixels and divide by 9," an ordinary average.

Suppose a scanner noise spike has left one pixel far too dark in an otherwise uniform bright patch:

200 200 200
200  50 200
200 200 200

To blur the center pixel, sum all 9 values and divide by 9. The eight border pixels are each 200, and the center is 50: sum = 200×8 + 50 = 1600 + 50 = 1650. Divide by 9: 1650 ÷ 9 = 183.33, which truncates to 183 using integer division. The noise spike of 50 has been smoothed to 183 — far closer to its true surroundings, while a genuine object edge (where neighboring pixels are legitimately very different) would only be softened, not erased, because in that case a larger fraction of the neighborhood would still agree with the differing pixel.

def box_blur_center(patch):
    total = 0
    for row in patch:
        for pixel in row:
            total += pixel
    return total // 9

noisy_patch = [
    [200, 200, 200],
    [200,  50, 200],
    [200, 200, 200]
]

print(box_blur_center(noisy_patch))  # 183

Trace it: the nested loops visit all 9 pixels in row-major order (200, 200, 200, 200, 50, 200, 200, 200, 200), adding each to total, which ends at 1650. 1650 // 9 is integer (floor) division: 9 × 183 = 1647, leaving a remainder of 3 that is discarded, so the function returns 183 — matching the hand calculation exactly. In a full-size image, this same 3×3 window would slide across every pixel position, one step at a time, recomputing a fresh average at each stop — that sliding motion is precisely what "convolving a kernel across an image" means.

Edge Detection: A Filter Built From Differences

Not every kernel has equal, positive weights. Detecting edges — the boundaries between light and dark regions, useful for finding the outline of a bubble, a printed character, or an object in a photo — uses a kernel with both positive and negative weights, because an edge is fundamentally about difference, not brightness.

Here is the simplest possible edge detector, working along a single row of pixels: for each pixel, subtract the pixel two positions to its left from the pixel two positions to its right. In symbols, if p is the row of pixel values, the edge response at position i is p[i+1] - p[i-1].

Take a row crossing from a dark region into a light region, like the edge of a printed letter on a page:

index:   0    1    2    3    4    5
value:  20   20   20  220  220  220

Compute the edge response at each interior position: at i=1, p[2]-p[0] = 20-20 = 0 — flat region, no edge. At i=2, p[3]-p[1] = 220-20 = 200 — a huge jump, correctly flagging that an edge lies right here. At i=3, p[4]-p[2] = 220-20 = 200 — still detecting the same edge from the other side. At i=4, p[5]-p[3] = 220-220 = 0 — flat again, on the light side. The response is exactly zero everywhere the image is uniform, and spikes to a large value exactly where the brightness changes — which is precisely what an edge detector is supposed to do.

def detect_edges_1d(row):
    edges = []
    for i in range(1, len(row) - 1):
        edges.append(row[i + 1] - row[i - 1])
    return edges

row = [20, 20, 20, 220, 220, 220]
print(detect_edges_1d(row))  # [0, 200, 200, 0]

Trace it: range(1, 5) produces i = 1, 2, 3, 4 (since len(row) is 6, so len(row)-1 is 5, and range stops before its second argument). For each, the function appends row[i+1] - row[i-1], giving 0, 200, 200, 0 in that order — matching the hand calculation. Notice too that the sign carries information: a rising edge (dark to light, left to right) gives a positive number, and a falling edge (light to dark) gives a negative number, so you can tell not just where an edge is but which direction the brightness is changing. Real image libraries extend this exact idea to two dimensions using a 3×3 kernel called the Sobel operator (named after computer scientist Irwin Sobel), which detects edges running in any direction across a full 2D image rather than just along one row — the underlying arithmetic of subtracting one side's neighbors from the other's is identical to what you just traced by hand.

Two Misconceptions Worth Correcting

Misconception 1: "A filter just recolors each pixel, like an Instagram filter." Many popular photo-editing "filters" do apply a simple per-pixel color remap — that is a point filter, and it's real, but it's only half the story. Genuine neighborhood filters like blur, sharpen, and edge detection cannot work pixel-by-pixel at all, because their entire purpose is to combine information from a pixel's surroundings. If you tried to blur an image using only each pixel's own value in isolation, you would get back the exact same image you started with — blurring is meaningless without neighbors.

Misconception 2: "Pixel arithmetic always gives a valid answer automatically." As the wraparound example showed, 240 + 20 does not become 255 by magic — computer arithmetic on a fixed-size integer type does exactly what you tell it to, including producing nonsense like 4 instead of 260 if you forget to clip the result. Every point filter that adds, subtracts, or scales pixel values needs an explicit clamping step; the computer will not enforce the 0–255 rule for you unless your code does.

Where This Shows Up Beyond the Classroom

The two ideas in this chapter — thresholding (a point filter) and convolution with a kernel (a neighborhood filter) — are not toy examples invented for this lesson. Aadhaar-based biometric authentication systems threshold and clean up fingerprint and iris images before matching them. ISRO's Earth-observation satellites, such as those in the Cartosat and RISAT series, apply neighborhood filters to raw satellite imagery to reduce sensor noise and sharpen boundaries between land features before the images are used for mapping or disaster monitoring. The arithmetic is exactly what you traced by hand above, just running over millions of pixels instead of a handful.

Practice: Test Your Understanding

1. Q1. A scanner reads this 2×2 patch over a bubble: [[10, 250], [250, 10]]. Using a threshold of 128 (filled if average < 128), is this bubble read as filled or unfilled?
   Answer: Sum = 10+250+250+10 = 520. Average = 520 ÷ 4 = 130. Since 130 is not less than 128, the scanner reads this as unfilled — even though half the patch is very dark, the average lands just above the cutoff. This shows why threshold-based filters can misclassify patches that mix very dark and very light pixels, which is why real OMR systems typically use a smaller, more centered patch.

2. Q2. Apply a 3×3 box blur to the center of [[100,100,100],[100,250,100],[100,100,100]].
   Answer: Sum = 100×8 + 250 = 800 + 250 = 1050. 1050 ÷ 9 = 116.67, which is 116 under integer division. The bright spike of 250 is smoothed down to 116.

3. Q3. Apply the 1D edge detector (p[i+1] - p[i-1]) to the row [50, 50, 180, 180, 50, 50] for all valid interior positions.
   Answer: i=1: 180-50=130. i=2: 180-50=130. i=3: 50-180=-130. i=4: 50-180=-130. Result: [130, 130, -130, -130]. The positive values mark a rising (dark-to-light) edge and the negative values mark a falling (light-to-dark) edge — this row contains a bright stripe between two dark regions, and the filter correctly finds both of its boundaries.

4. Q4. Why can a point filter never blur an image, no matter how it is written?
   Answer: A point filter computes each output pixel from that single pixel's own value alone. Blurring requires mixing a pixel's value with its neighbors' values, so the computation needs access to more than one input pixel per output pixel — which by definition only a neighborhood (convolution) filter provides.

Summary

A digital image is a grid of numbers: one 0–255 value per pixel in grayscale, or three such values (R, G, B) per pixel in color, combined using weights like 0.299/0.587/0.114 to get an accurate grayscale equivalent. Every image filter falls into one of two families. Point filters, like thresholding an OMR bubble or brightening a photo, compute each output pixel from that one pixel's input value alone, and must explicitly clip results back into the 0–255 range to avoid overflow bugs. Neighborhood filters, like box blurring and edge detection, slide a small weighted grid called a kernel across the image and combine each pixel with its surroundings — equal positive weights average and smooth, while paired positive-and-negative weights subtract neighboring regions from each other to expose sudden brightness changes as edges. Every one of these operations, from the crude 4×4 patches you worked through by hand here to the millions of pixels in a satellite photograph, is built from nothing more exotic than addition, division, and comparison.

Think About It

Think about this: How would you explain image processing: pixels and filters to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.