Data Preprocessing: Handling Missing Values and Outliers

The Listing With a Suspicious Address

You are building a price predictor for 2BHK flats in a Bengaluru neighbourhood. You scrape eleven listings. Nine have a price. Two show a blank cell — the broker forgot to fill it in, or the listing was pulled before the price was updated. Of the nine that do have a price, eight cluster in a believable band, roughly ₹52 lakh to ₹85 lakh. The ninth reads ₹250 lakh, for a flat with the same square footage, same floor, same locality as the rest.

Before you can train anything on this data, you have to answer two separate questions, and it matters which one you answer first. What do you write into the two blank cells? And what do you do with the flat that costs three times what its neighbours cost — is it a mansion-grade renovation someone genuinely paid for, or a keystroke that turned "25" into "250"? Get either answer wrong and every statistic you compute afterwards — mean, variance, a regression coefficient, a distance in k-NN — inherits the mistake silently. This chapter builds the machinery to answer both questions with arithmetic, not guesswork, and shows exactly how much a careless answer costs you, in numbers you can check by hand.

Why Data Goes Missing: Three Different Reasons

Before touching a blank cell, ask why it's blank — because the right fix depends entirely on the mechanism, and a fix that's correct for one mechanism actively damages your model under a different one. Statisticians split missingness into three categories.

• MCAR — Missing Completely At Random. The chance a value is missing has nothing to do with the value itself or with anything else you measured. A temperature-logging sensor drops one reading in ten thousand because of a momentary Wi-Fi glitch, unrelated to whether that reading would have been hot or cold. Under MCAR, the rows you do have are a fair, unbiased sample of the rows you don't — deleting them or imputing them causes no systematic distortion, only some loss of precision.
• MAR — Missing At Random. The chance of being missing depends on another variable you did observe, not on the missing value itself. In a school health survey, height and weight are missing more often for students who were absent on measurement day — and absence correlates with the class section, which you do have on record. Once you account for class section, the missingness is "random" again. Fixing this properly usually means imputing within groups (per class), not with one global number.
• MNAR — Missing Not At Random. The chance of being missing depends on the unobserved value itself. In an income survey, very high earners are disproportionately the ones who decline to answer the income question. You cannot see the value that's causing its own absence — no imputation trick recovers it, because the standard imputation methods below all assume the missing values would look statistically similar to the ones you kept, and that assumption is exactly what MNAR breaks.

Our two blank flat prices are almost certainly MCAR or MAR — a broker's oversight has no reason to correlate with the flat being cheap or expensive. That's what licenses everything we do with them in the next section. If a real dataset's missingness looks MNAR (say, sellers of overpriced flats quietly withdrawing the listing before the price is finalised), no formula in this chapter is the right tool — you'd need to model the missingness itself, which is a research-level problem outside this chapter's scope.

Finding and Counting the Gaps

In pandas, a missing numeric entry is represented as NaN (Not a Number), and every cell in a DataFrame can be tested for it independently of what type of value it should contain. Let's put our eleven flats into a DataFrame:

import pandas as pd
import numpy as np

flats = pd.DataFrame({
    'flat_id': ['F1','F2','F3','F4','F5','F6','F7','F8','F9','F10','F11'],
    'price_lakh': [52, 58, 61, 65, 68, 72, 78, 85, 250, np.nan, np.nan]
})

print(flats['price_lakh'].isnull().sum())

isnull() returns a Boolean Series, True wherever a cell is NaN; .sum() treats True as 1 and adds them up. The output is 2 — exactly F10 and F11, our two blank listings. This one-liner is the first thing you should run on any dataset before doing anything else with it; silently proceeding with hidden NaNs will make later arithmetic (a mean, a distance, a matrix multiply) return NaN for the whole computation, not just the affected row.

Two Ways to Handle a Gap: Delete or Impute

The blunt option is dropna() — throw away any row with a missing value:

clean = flats.dropna(subset=['price_lakh'])
print(len(flats), len(clean))

Output: 11 9. Two rows gone. Deletion is only defensible under MCAR, and only when the fraction missing is small — here it's 2/11 ≈ 18%, already large enough that throwing the rows away wastes real information you paid to collect (someone photographed those flats, negotiated a viewing, logged square footage). The alternative is imputation: filling the gap with a value computed from the data you do have — commonly the mean, median, or (for categorical data) the mode of the column.

Which of mean or median should we use here? This is not a coin flip — our outlier at ₹250 lakh answers it for us:

known = flats['price_lakh'].dropna()
print(known.mean(), known.median())

Trace it: the nine known prices sorted are 52, 58, 61, 65, 68, 72, 78, 85, 250. Their sum is 52+58+61+65+68+72+78+85+250 = 789, and 789/9 = 87.666… The median of nine sorted values is the 5th one: 68. Output: 87.66666666666667 68.0. The single ₹250 lakh flat has dragged the mean nearly ₹20 lakh above where eight of the nine listings actually sit — the mean is not "typical" here, it's being pulled by one point. The median, by construction, only cares about rank order, so the ₹250 lakh flat contributes exactly the same "one vote" as any other flat regardless of how far out it sits. When your column contains — or might contain — an outlier, median imputation is the safer default, precisely because it doesn't inherit the outlier's distortion. We impute with 68, not 87.67:

flats['price_imputed'] = flats['price_lakh'].fillna(known.median())

The Hidden Cost of Mean Imputation: A Variance Derivation

Even when there's no outlier around to make the choice obvious, imputation is not the "free" fix it looks like. Filling in a gap with the mean silently shrinks the variance of your column — and you can derive exactly how much, from first principles.

Let a population of n values have mean μ = (1/n)Σxᵢ and (population) variance σ² = (1/n)Σ(xᵢ − μ)². Suppose m of the n values are missing; call the set of their indices M. The remaining n − m observed values have their own mean, x̄₀ = (1/(n−m))·Σ_i∉M xᵢ. Mean imputation builds a new dataset y where yᵢ = xᵢ for every observed index, and yᵢ = x̄₀ for every missing index.

Step 1 — the new mean. μ′ = (1/n)Σyᵢ = (1/n)[Σ_i∉Mxᵢ + m·x̄₀]. By definition Σ_i∉Mxᵢ = (n−m)x̄₀, so μ′ = (1/n)[(n−m)x̄₀ + m·x̄₀] = (1/n)(n·x̄₀) = x̄₀. Imputing with the observed mean leaves the overall mean exactly unchanged — no surprise, that's the whole point of choosing the mean as the fill value.

Step 2 — the new variance. σ′² = (1/n)Σ(yᵢ − μ′)² = (1/n)Σ(yᵢ − x̄₀)². Split the sum over observed and missing indices. For missing indices, yᵢ − x̄₀ = x̄₀ − x̄₀ = 0, contributing nothing. So:

σ'² = (1/n) · Σ_{i not in M} (xᵢ − x̄₀)²

Now define s₀² = (1/(n−m))·Σ_i∉M(xᵢ − x̄₀)² — the population variance of the observed subset alone. Then Σ_i∉M(xᵢ − x̄₀)² = (n−m)·s₀², and substituting:

σ'² = (n − m)/n · s₀²  =  (1 − m/n) · s₀²

This is an exact identity, not an approximation — it relates the new variance directly to the variance of whichever values you happened to keep. The popular rule of thumb "mean imputation shrinks variance by roughly a factor of (1 − m/n)" comes from one further, non-exact step: assuming the retained subsample's variance s₀² is close to the true population variance σ² (i.e., the m values you lost happen to be statistically unremarkable, not disproportionately the extreme ones). When that assumption holds, σ′² ≈ (1 − m/n)·σ². When it doesn't — for instance if the missing values happened to be unusually close to the mean, leaving a more spread-out remainder — s₀² can differ from σ² by a wide margin, especially for small samples where a subsample of size n−m carries its own sampling variability. The exact relationship is always in terms of s₀²; the σ²-based version is a convenient estimate you should sanity-check, not trust blindly.

Checking the Formula Against Real Numbers

Let's verify this on the eight flats whose prices we're not in any doubt about — F1 through F8, deliberately set aside from the ₹250 lakh outlier so the outlier doesn't contaminate a demonstration about missingness:

normal8 = pd.Series([52, 58, 61, 65, 68, 72, 78, 85])
print(normal8.mean(), normal8.var(ddof=0))

Sum = 52+58+61+65+68+72+78+85 = 539, mean = 539/8 = 67.375. Sum of squared deviations works out to 815.875 (each term: (52−67.375)² = 236.390625, (58−67.375)²=87.890625, (61−67.375)²=40.640625, (65−67.375)²=5.640625, (68−67.375)²=0.390625, (72−67.375)²=21.390625, (78−67.375)²=112.890625, (85−67.375)²=310.640625; these sum to 815.875). Divide by n=8: σ² = 101.984375. Output: 67.375 101.984375.

Now pretend F2 (58) and F7 (78) are missing (m=2, n=8), and mean-impute them:

observed = pd.Series([52, 61, 65, 68, 72, 85])
x0 = observed.mean()
print(x0, observed.var(ddof=0))

Sum of the six retained values = 52+61+65+68+72+85 = 403, so x̄₀ = 403/6 = 67.1667. Their squared deviations from 67.1667 sum to 614.8333, giving s₀² = 614.8333/6 = 102.4722 — output 67.16666666666667 102.47222222222223. Notice s₀² (102.47) lands almost exactly on σ² (101.98): F2 and F7 were deliberately chosen because their own squared deviations from the mean (87.89 and 112.89) straddle the population average squared deviation (101.98), so removing them doesn't skew the remaining spread much either way — this is what "a representative pair to remove" looks like numerically.

imputed8 = pd.Series([52, 61, 65, 68, 72, 85, x0, x0])
print(imputed8.var(ddof=0))
print(0.75 * normal8.var(ddof=0))

The exact formula predicts σ′² = (1 − 2/8)·s₀² = 0.75 × 102.4722 = 76.8542 — and that is exactly what imputed8.var(ddof=0) returns: 76.85416666666667. Compare this to the σ²-based estimate, 0.75 × 101.984375 = 76.48828125: the two are within 0.37 of each other, a relative gap under half a percent — because here s₀² really was close to σ². Overall, variance fell from 101.98 to 76.85, a 24.6% reduction, essentially matching the 25% the (1 − m/n) factor predicts. In standard-deviation terms the shrinkage is gentler — SD falls by a factor of √0.75 ≈ 0.866, i.e. about 13%, since variance shrinks as the square of the SD's shrinkage. The lesson to keep is structural, not the specific 24.6%: mean imputation always removes exactly the spread that the imputed points would have contributed, and the size of that loss scales with m/n — impute 2 values out of 1,000 and the effect is negligible; impute 200 out of 1,000 and a fifth of your variance has quietly vanished, understating every downstream confidence interval and standard error computed from that column.

Back to Our Flats: Formal Outlier Detection

We already used common sense to spot ₹250 lakh as suspicious. Now let's detect it with a rule that would work even if we didn't already know which value was strange, using the median-imputed, full eleven-value column: 52, 58, 61, 65, 68, 68, 68, 72, 78, 85, 250 (sorted; the three 68s are F5's genuine price plus the two median-imputed gaps).

Method 1 — the IQR fence. Sort the data and find the first quartile Q1 (25th percentile) and third quartile Q3 (75th percentile); their gap, IQR = Q3 − Q1, measures the spread of the "middle half" of the data, untouched by extreme values. Any point further than 1.5×IQR below Q1 or above Q3 is flagged.

full = flats['price_lakh'].fillna(known.median())
q1 = full.quantile(0.25)
q3 = full.quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5*iqr
upper = q3 + 1.5*iqr
print(q1, q3, iqr, lower, upper)

Pandas' default quantile interpolation places the 11 sorted values at positions 0–10, and for the 25th percentile computes position (11−1)×0.25 = 2.5 — halfway between index 2 (value 61) and index 3 (value 65), giving Q1 = 61 + 0.5×(65−61) = 63. For the 75th percentile, position (11−1)×0.75 = 7.5, halfway between index 7 (72) and index 8 (78): Q3 = 72 + 0.5×(78−72) = 75. So IQR = 75 − 63 = 12, lower fence = 63 − 1.5×12 = 45, upper fence = 75 + 1.5×12 = 93. Output: 63.0 75.0 12.0 45.0 93.0.

outliers = full[(full < lower) | (full > upper)]
print(outliers)

Every known price sits inside [45, 93] except one: F9 at 250, far past the upper fence of 93. Output is a one-row Series, index 8, value 250.0. The IQR method caught it cleanly, and — notice — it did so using only Q1 and Q3, neither of which the ₹250 lakh flat had any influence over, since it sits well outside the middle 50% of the data to begin with.

Method 2 — Z-Scores, and Their Blind Spot

The more familiar rule flags any point whose z-score, z = (x − μ)/σ, exceeds 3 in magnitude — "more than three standard deviations from the mean."

mean = full.mean()
std = full.std(ddof=0)
z_scores = (full - mean) / std
print(round(mean, 2), round(std, 2), round(z_scores.iloc[8], 2))

Sum of the eleven values (52+58+61+65+68+68+68+72+78+85+250) = 925, mean = 925/11 = 84.09. The sum of squared deviations from 84.09 works out to 3,762,484/121 ≈ 31,094.9, and dividing by n=11 gives σ² ≈ 2826.81, so σ ≈ 53.17. The outlier's z-score is (250 − 84.09)/53.17 ≈ 3.12. Output: 84.09 53.17 3.12. It clears the usual threshold of 3 — but only barely, and that's the method's weak point: the outlier inflates the very standard deviation being used to detect it. A single point 166 units from the mean contributes a squared deviation of about 27,526 to a total of about 31,095 — over 88% of the entire variance is coming from the one point the test is trying to flag. Add a second flat priced even further out and the z-score for both points could shrink even as the data gets more obviously contaminated, because σ grows faster than the numerator. This self-masking effect is a genuine limitation of mean-and-SD-based detection, not a minor technicality.

Method 3 — The Modified Z-Score (Median Absolute Deviation)

The fix is to replace both the mean and the SD with their robust counterparts: the median, and the median absolute deviation (MAD) — the median of |xᵢ − median|. Neither statistic moves much when one point is dragged far away, because both only depend on rank, not distance.

median = full.median()
mad = (full - median).abs().median()
modified_z = 0.6745 * (full - median) / mad
print(median, mad, round(modified_z.iloc[8], 2))

The median of the eleven sorted values is the 6th one: 68. Absolute deviations from 68 are 16, 10, 7, 3, 0, 0, 0, 4, 10, 17, 182; sorted, the middle (6th) value is 7 — so MAD = 7. The constant 0.6745 rescales MAD so that, for a perfectly normal distribution, the modified z-score matches an ordinary z-score in magnitude (0.6745 is the 75th percentile of the standard normal distribution, chosen because MAD itself, for a normal distribution, converges to about 0.6745σ). For F9: modified z = 0.6745 × (250 − 68)/7 = 0.6745 × 26 = 17.54. Output: 68.0 7.0 17.54. Compare the two detectors side by side: ordinary z-score = 3.12 (barely past the usual cutoff of 3), modified z-score = 17.54 (obliterating the usual cutoff of 3.5). The modified score isn't just "more sensitive" by chance — it's immune to the exact distortion that weakened the first method, because neither the median nor the MAD used 250 to compute anything except its own deviation. For any dataset where a genuine outlier might be large enough to warp the mean and SD, the modified z-score is the more trustworthy instrument, and this is precisely why: the statistic you use to judge the outlier should not itself be built out of the outlier.

Misconception: "Just Delete the Outlier"

A common shortcut, once a point is flagged, is to drop it and move on. This is wrong often enough that you should treat it as a decision requiring evidence, not a reflex. An outlier is a statistical label — "far from the rest of the data" — and says nothing on its own about why the point is far away. There are at least three distinct causes, and only one of them justifies deletion:

• Data-entry or measurement error (an extra zero, a unit mismatch, a sensor fault) — deletion, or correction if the true value is recoverable, is appropriate.
• A genuine, rare event (a real penthouse with a terrace garden that legitimately sold for ₹2.5 crore) — deleting it doesn't clean your data, it deletes real information and teaches your model that such flats can't exist, which will make it fail exactly when a genuine premium listing shows up in production.
• A different underlying population mixed into your sample (a commercial unit accidentally scraped alongside residential 2BHKs) — the fix is to separate the populations, not to silently prune one point and treat the rest as homogeneous.

For our F9, ₹250 lakh for a plain 2BHK in a locality where eight comparable listings sit between ₹52 lakh and ₹85 lakh is far more consistent with a data-entry slip (perhaps ₹25.0 lakh, misread or mistyped as ₹250) than with a legitimate luxury listing — you'd expect a genuinely premium flat to also differ in square footage, amenities, or floor, which our scenario doesn't show. The right move here is to go back to the source listing and check, and only delete or correct once you've confirmed which of the three causes applies — not before. Reflexive deletion is a misconception precisely because it substitutes a rule ("if flagged, remove") for the investigation the flag was supposed to trigger.

Which Comes First: Imputation or Outlier Detection?

Our workflow imputed the two missing prices before formally testing for outliers — is that the right order? The general principle: run whichever step first uses statistics that the other step's contamination can't distort. We imputed with the median of the known values, and the median is a robust statistic — the presence of the ₹250 lakh outlier among the nine known values didn't change what got substituted into F10 and F11 (compare: had we imputed with the mean instead, we'd have written 87.67 into both blanks, and that number would have carried the outlier's distortion into two more rows). Because our imputation step was already robust to the outlier, doing it first caused no damage, and it let us run outlier detection afterwards on a complete, gap-free column. Had we instead planned to mean-impute, the safer order would be to detect and resolve the outlier first, then compute a mean that isn't corrupted by it, and only then fill the gaps. As a rule of thumb: robust-statistic imputation (median, mode) is order-independent with outlier handling; mean-based imputation is not, and should follow outlier resolution, not precede it.

A Picture of the Full Dataset

Where This Sits in Your Syllabus

In CBSE's Artificial Intelligence curriculum, this chapter's work is exactly the "Data Exploration" stage of the AI Project Cycle you met when problem-scoping was introduced — the stage that sits between acquiring data and building a model, and the stage examiners most often test with "identify the missingness type and justify your imputation choice" style questions. The variance derivation leans on material you'll meet formally in the CBSE Class 11 Statistics chapter — mean, variance and standard deviation of ungrouped data, and quartile deviation — both are JEE Main and BITSAT syllabus topics under "Statistics," so the algebra above is not a detour, it's the same formula you'll be asked to apply in a board or entrance exam, just applied here to a dataset with a story attached instead of a bare list of numbers. The IQR fence rule and z-score/modified z-score comparison don't appear on JEE directly (JEE's statistics questions stay at mean/variance/quartile level), but they're standard vocabulary in any applied-statistics or introductory data-science assessment, including GATE's foundational probability-and-statistics questions.

Practice

1. A population of n = 12 values has variance σ² = 64. If m = 3 values are mean-imputed, and the retained subsample's variance is close to σ², what does the exact formula predict for the new variance σ′²?
2. Suppose we had imputed F10 and F11 with the mean of the known prices (87.67) instead of the median (68). Recompute the median of the full eleven-value column with 87.67 substituted twice. Sort the new list and pick the 6th value.
3. A thirteenth flat is listed at ₹40 lakh. Using the fences already computed (LF = 45, UF = 93), is it flagged as a statistical outlier? What real-world explanations would you check before deciding whether to delete, correct, or keep it?
4. Using median = 68 and MAD = 7 from the full dataset, compute the modified z-score for the flat priced at ₹52 lakh. Is it flagged at the usual threshold of 3.5?

Answer check. (1) σ′² = (1 − 3/12) × 64 = 0.75 × 64 = 48. (2) Sorted: 52, 58, 61, 65, 68, 72, 78, 85, 87.67, 87.67, 250 — the 6th of 11 values is 72, so the median shifts from 68 to 72 purely because of the imputation method chosen, even though not a single "real" price changed — this is the imputation-choice effect the median-vs-mean discussion warned about. (3) 40 < 45, so yes, it's flagged on the low side; before acting, check whether it's a smaller-carpet-area unit mislabelled as the same configuration, a distress sale, or a genuine data-entry issue — the fence tells you where to look, not what you'll find. (4) M = 0.6745 × (52 − 68)/7 = 0.6745 × (−16/7) ≈ −1.54; since |−1.54| < 3.5, it is not flagged — consistent with 52 sitting comfortably inside the fences on the boxplot.

Summary

Missing data has a mechanism — MCAR, MAR, or MNAR — and the mechanism, not habit, should decide whether you delete or impute, and whether you impute globally or within subgroups. Mean imputation is not statistically free: it provably shrinks variance by an exact factor of (n−m)/n applied to the retained subsample's own variance, a fact you derived from the definitions of mean and variance rather than took on faith, and verified to within half a percent on eight real numbers. When outliers are present or suspected, median imputation avoids inheriting their distortion, as the ₹87.67-lakh-vs-₹68-lakh comparison showed directly. Outlier detection has multiple tools with different failure modes: the IQR fence is robust and simple; the ordinary z-score can be weakened by the very outlier it's hunting, because the outlier inflates its own denominator; the modified z-score, built from median and MAD, sidesteps that trap entirely. And an outlier is a statistical flag, not a verdict — the correct response depends on whether you're looking at an error, a rare-but-real event, or a mixed population, and only the first of those three justifies deleting the point outright.

Think About It

Think about this: How would you explain data preprocessing: handling missing values and outliers to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.