Building a Complete Data Preprocessing Pipeline

Here is a question that quietly breaks a lot of machine learning models before a single algorithm has even run: a bank wants to predict whether a loan should be approved, using two features about the applicant — Age (in years) and Annual Family Income (in rupees). Two past applicants are on record. Applicant P1, age 15, income ₹450,000, was approved. Applicant P2, age 23, income ₹480,000, was rejected. A new applicant Q arrives: age 16, income ₹550,000. Using a k-nearest-neighbours (KNN) classifier with k = 1, which past applicant is Q "closer" to?

Age says the answer is obvious — Q is 16, only one year from P1's 15, and seven years from P2's 23. Income-wise, Q sits between the two, roughly ₹100,000 above P1 and ₹70,000 above P2. Every instinct says P1 is the better match. Now compute the actual Euclidean distance that KNN uses, in raw units:

distance(Q, P1) = √[(16−15)² + (550000−450000)²]
                 = √[1 + 10,000,000,000] ≈ 100,000.00

distance(Q, P2) = √[(16−23)² + (550000−480000)²]
                 = √[49 + 4,900,000,000] ≈ 70,000.00

Because 70,000 < 100,000, plain KNN says Q is closer to P2 and predicts "rejected" — the opposite of what age similarity suggested. Nothing is wrong with the KNN algorithm. The problem is that Age lives on a scale of tens (typical spread of a couple of years) while Income lives on a scale of hundreds of thousands. In a Euclidean distance formula, whichever feature has the larger numbers automatically wins, regardless of which feature actually carries more predictive information. This is not a rare edge case — it is the default behaviour of every distance-based, gradient-based, or regularised algorithm (KNN, K-means, SVM, PCA, linear/logistic regression with regularisation, neural networks) whenever raw features are fed in on mismatched scales.

This chapter is about the fix: a data preprocessing pipeline, a fixed, repeatable sequence of transformations — handling missing values, detecting outliers, scaling features, and encoding categories — applied consistently to training and test data so that the numbers a model sees actually reflect the patterns in the data, not accidents of measurement units. By the end, you will see exactly how standardising Age and Income flips the verdict above, and you will build the whole pipeline in code using scikit-learn, the library used throughout real ML workflows and referenced in CBSE's Artificial Intelligence curriculum.

The Four Stages, and the One Rule That Governs All of Them

A preprocessing pipeline has four standard stages, always applied in this order:

1. Handle missing values — fill in or remove gaps in the data (imputation).
2. Detect and handle outliers — flag values so extreme they would distort statistics or model training.
3. Scale numeric features — put every numeric column on a comparable range, exactly like the Age/Income example demands.
4. Encode categorical features — convert text categories (like "City") into numbers a model can use, without inventing a false ordering.

Every one of these stages is governed by a single non-negotiable rule, which we will justify carefully once we reach the code: a pipeline's statistics — the mean used to fill a gap, the quartiles used to catch an outlier, the mean/standard-deviation used to scale, the set of categories used to encode — must be learned only from the training data, then applied unchanged to the test data. Learn them from the full dataset (train and test combined) and you leak information about the test set into training, producing a model that looks better in your evaluation than it will ever perform in reality. Keep this rule in your head as you read; we return to it explicitly in the final section.

Stage 1: Handling Missing Data

Real datasets have gaps — a sensor failed to log a reading, a survey respondent skipped a question, a field was entered as blank. Two of the most common fixes are mean imputation (replace the missing value with the column's average) and median imputation (replace it with the column's middle value). They are not interchangeable, and choosing wrong is a classic, gradable mistake.

Suppose you are recording the monthly pocket money (in ₹) of seven students, and one entry is missing:

500, 600, 550, 580, 620, ?, 5000

One student comes from a very high-income family and receives ₹5000/month — a real value, not an error, but far outside the range of the other six. To impute the missing value:

Mean of the known six values: (500+600+550+580+620+5000) / 6 = 7850 / 6 = ₹1308.33.

Median of the known six values: sorted, they are 500, 550, 580, 600, 620, 5000. With six (even) values, the median is the average of the 3rd and 4th: (580+600)/2 = ₹590.

The mean, ₹1308.33, does not resemble pocket money for any student in this data — it is a number the ₹5000 outlier single-handedly dragged upward, sitting more than double the highest of the six ordinary values. The median, ₹590, sits right where five of the six known values cluster and is a far more honest guess for what the missing student's pocket money probably was. This is the general rule: use the mean when the column is roughly symmetric with no extreme values; use the median when the column is skewed or contains outliers, because the median is far less sensitive to extreme values than the mean (moving the 5000 to 50,000 would barely move the median but would send the mean soaring).

Common misconception: many students assume mean imputation is always "more accurate" because it uses more arithmetic. It is not — accuracy here means "how representative is the filled-in value," and on skewed data the median wins decisively, as the numbers above show.

Stage 2: Detecting Outliers — The Interquartile Range (IQR) Method

An outlier is a value so far from the bulk of the data that it likely distorts any statistic computed from the dataset (as the ₹5000 pocket-money entry nearly did to the mean above). The standard, exam-relevant method for flagging outliers uses quartiles.

Sort the data. The median (Q2) splits it into a lower half and an upper half. The first quartile (Q1) is the median of the lower half; the third quartile (Q3) is the median of the upper half. The interquartile range is IQR = Q3 − Q1, the spread of the "middle 50%" of the data, deliberately ignoring the extremes. Tukey's rule then defines the fences:

Lower fence = Q1 − 1.5 × IQR
Upper fence = Q3 + 1.5 × IQR

Any value outside these fences is flagged as an outlier. Note the convention used throughout this chapter for splitting the data: when the total count n is odd, the median itself is excluded from both the lower and upper halves before taking their medians (this matches the method used in the NCERT statistics chapter and is the version examiners expect); when n is even, the data splits cleanly into two equal halves with no value to exclude.

Worked example (even n). Monthly family income (₹) recorded for 8 students in a scholarship-eligibility study, already sorted:

18000, 24000, 34000, 40000, 40500, 41000, 44000, 95000

n = 8 is even, so the lower half is the first four values (18000, 24000, 34000, 40000) and the upper half is the last four (40500, 41000, 44000, 95000).

Q1 = median(18000, 24000, 34000, 40000) = (24000+34000)/2 = 29000
Q3 = median(40500, 41000, 44000, 95000) = (41000+44000)/2 = 42500
IQR = 42500 − 29000 = 13500

Lower fence = 29000 − 1.5(13500) = 29000 − 20250 = 8750
Upper fence = 42500 + 1.5(13500) = 42500 + 20250 = 62750

Every value lies between 8750 and 62750 except ₹95,000, which exceeds the upper fence — it is flagged as an outlier and would typically be investigated (a genuine high-income family) rather than blindly deleted, since deleting real data changes the population you are studying.

A quiet trap when moving from hand calculation to code. If you compute quartiles on this exact dataset using a statistics library's default method (linear interpolation between ranks, which is what pandas.Series.quantile() and numpy.quantile() use by default), you get Q1 = 31500 and Q3 = 41750 instead of 29000 and 42500 — different numbers from the same data, because "quartile" does not have one universal definition; different interpolation rules produce different (though similar) answers. For CBSE board answers, use the hand method shown above (median-of-halves); when you write code, know that the library's default will differ slightly and that is expected, not a bug.

Stage 3: Feature Scaling — Min-Max Normalization and Z-score Standardization

Now we return to the Age/Income problem from the opening and fix it properly, deriving the two standard scaling formulas rather than just quoting them.

Min-max normalization rescales every value in a column to fall inside [0, 1]:

X' = (X − min) / (max − min)

Why this works: when X = min, X' = 0/(max−min) = 0. When X = max, X' = (max−min)/(max−min) = 1. Every value in between lands proportionally between 0 and 1. This is simple and guarantees a fixed range, but it is fragile — a single extreme outlier stretches max (or shrinks min) and compresses every other value toward one end of the [0, 1] range.

Z-score standardization instead rescales using the column's mean μ and standard deviation σ:

Z = (X − μ) / σ

To see exactly what this does to a column's shape, we need two standard results about how mean and variance behave under a linear transformation Y = aX + b, and it is worth deriving both rather than accepting them on faith, since the whole justification for z-scores rests on them.

Claim 1: E[aX + b] = a·E[X] + b. This follows directly from linearity of expectation: E[aX + b] = a·E[X] + E[b] = a·E[X] + b (a constant's expected value is itself).

Claim 2: Var(aX + b) = a²·Var(X). By definition, Var(Y) = E[(Y − E[Y])²]. Substitute Y = aX + b and E[Y] = aE[X] + b from Claim 1:

Var(aX+b) = E[(aX+b − (aE[X]+b))²]
          = E[(aX − aE[X])²]
          = E[a²(X − E[X])²]
          = a² · E[(X − E[X])²]
          = a² · Var(X)

Adding a constant b shifts every value equally, so it cannot change how spread out they are — hence b disappears from the variance, which is exactly what the algebra shows.

Now apply both claims to Z = (X − μ)/σ, which is a linear transform with a = 1/σ and b = −μ/σ:

E[Z] = (1/σ)·μ − μ/σ = 0
Var(Z) = (1/σ)² · Var(X) = (1/σ²) · σ² = 1

So standardization always produces a column with mean exactly 0 and standard deviation exactly 1, regardless of the original units — years, rupees, kilometres, or anything else all end up on the same dimensionless scale. This is precisely the property that fixes the Age/Income problem.

Fixing the opening example. Suppose the bank's training data (across many applicants, not just P1/P2) has statistics μ_age = 17, σ_age = 2, μ_income = ₹500,000, σ_income = ₹250,000. Standardize all three points:

P1 (15, 450000): z_age = (15−17)/2 = −1.0,  z_income = (450000−500000)/250000 = −0.2
P2 (23, 480000): z_age = (23−17)/2 = 3.0,   z_income = (480000−500000)/250000 = −0.08
Q  (16, 550000): z_age = (16−17)/2 = −0.5,  z_income = (550000−500000)/250000 = 0.2

distance(Q, P1) = √[(−0.5−(−1.0))² + (0.2−(−0.2))²] = √[0.25+0.16] = √0.41 ≈ 0.64
distance(Q, P2) = √[(−0.5−3.0)² + (0.2−(−0.08))²] = √[12.25+0.0784] ≈ √12.33 ≈ 3.51

Once both features are on the same standardized scale, distance(Q, P1) ≈ 0.64 is far smaller than distance(Q, P2) ≈ 3.51 — the verdict flips to "approved," matching what age similarity suggested all along. Scaling did not change any underlying fact about the applicants; it corrected an artefact of measuring one feature in single digits and the other in hundreds of thousands.

Which scaler to use? Min-max is preferred when you need a strict, known [0, 1] range (for example, feeding pixel intensities into a neural network) and outliers have already been handled. Z-score standardization is preferred by default for distance-based and gradient-based algorithms (KNN, SVM, logistic/linear regression, PCA) because it is less distorted by a single extreme value, since μ and σ are less sensitive than a raw max to one outlier point (though still not immune — that is exactly why outlier handling in Stage 2 comes before scaling, not after).

Stage 4: Encoding Categorical Variables

Age and Income are numeric. But a feature like City (Chennai, Delhi, Mumbai) is categorical text, and every ML algorithm ultimately does arithmetic on numbers, not strings. The naive fix — assign Chennai = 0, Delhi = 1, Mumbai = 2 — is called label encoding, and for a category with no natural order (a nominal variable), it is a serious, commonly graded mistake.

Common misconception: label encoding "just turns text into numbers," so it must be harmless. It is not, because assigning 0, 1, 2 silently tells the model "Mumbai is numerically greater than Delhi, which is greater than Chennai," and that Chennai-to-Mumbai (distance 2) is twice as far as Chennai-to-Delhi (distance 1) — a completely fabricated ordering and false notion of magnitude that a distance-based or linear model will use as if it were real.

The fix is one-hot encoding: replace the single City column with one binary (0/1) column per category, so no ordering or magnitude is implied — each category is an independent yes/no dimension:

City       is_Chennai   is_Delhi   is_Mumbai
Chennai        1            0          0
Delhi          0            1          0
Mumbai         0            0          1
Delhi          0            1          0

The trade-off is dimensionality: a categorical column with k distinct categories becomes k numeric columns instead of one. For a feature like "state of India" (36 categories) that is a real cost, but it is a correct cost — unlike label encoding, it never invents a fake numeric relationship. (In some pipelines, one column is dropped, e.g. keeping only is_Delhi and is_Mumbai, since "not Delhi and not Mumbai" already implies Chennai; this avoids a mild redundancy called the dummy variable trap, relevant for linear regression, and is done automatically when you set drop='first' in scikit-learn's encoder.)

Assembling the Full Pipeline in scikit-learn

With all four stages understood individually, here is the one rule from Section 2 made concrete: split into train/test first, then fit every imputer, scaler, and encoder only on the training fold, and re-use those exact fitted values (not new ones) to transform the test fold. If you instead computed, say, the imputation median from the full dataset (train + test combined), the training process would have quietly "seen" a statistic derived partly from the test set — a leak that makes your reported test accuracy optimistic and unreliable as an estimate of real-world performance.

Consider 8 students with three numeric features (Age, FamilyIncome with one missing value, StudyHours, Attendance) and one categorical feature (City: Delhi or Mumbai), predicting whether they Pass:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

data = pd.DataFrame({
    "Age":          [15, 16, 17, 16, 15, 17, 16, 15],
    "FamilyIncome": [38000, 42000, np.nan, 55000, 29000, 61000, 47000, 33000],
    "StudyHours":   [12, 8, 15, 10, 6, 18, 11, 9],
    "Attendance":   [88, 72, 95, 80, 65, 97, 78, 70],
    "City":         ["Delhi","Mumbai","Mumbai","Delhi","Mumbai","Delhi","Delhi","Mumbai"],
    "Pass":         [1, 0, 1, 1, 0, 1, 0, 0]
})

X = data.drop(columns=["Pass"])
y = data["Pass"]

# Split BEFORE any statistic is computed — this is the leakage boundary
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

numeric_features = ["Age", "FamilyIncome", "StudyHours", "Attendance"]
categorical_features = ["City"]

numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipe = Pipeline([
    ("onehot", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("cat", categorical_pipe, categorical_features)
])

X_train_t = preprocessor.fit_transform(X_train)   # LEARNS stats from train only
X_test_t  = preprocessor.transform(X_test)         # REUSES those exact stats

Run this exact code and here is what happens, traced precisely. With test_size=0.25 on 8 rows and random_state=42, scikit-learn's split (which shuffles row indices, independent of the feature values) always produces the same partition for this size and seed: training rows are the original indices [0, 7, 2, 4, 3, 6] and test rows are [1, 5] — 6 rows train, 2 rows test. The single missing FamilyIncome value (original index 2) lands in the training fold, so SimpleImputer(strategy="median") computes its fill value from the five other training-fold incomes {38000, 33000, 29000, 55000, 47000}, whose median is ₹38,000 — and that fixed number (not a freshly recomputed one) is exactly what would be used if a row with a missing income appeared in the test fold too. Both cities, Delhi and Mumbai, happen to appear in the training fold, so OneHotEncoder learns two category columns from training data alone. preprocessor.fit_transform(X_train) therefore outputs a NumPy array of shape (6, 6) — 6 training rows × (4 standardized numeric columns + 2 one-hot city columns) — and preprocessor.transform(X_test), reusing the training fold's median, mean, standard deviation, and category list without recomputing any of them, outputs shape (2, 6) for the 2 test rows.

This is the entire discipline in one call pair: fit_transform exactly once, on the training fold; transform only, everywhere else, forever.

How This Maps to Your Exams

The statistics underneath Stages 1 and 2 — mean, median, quartiles, interquartile range, mean deviation, variance, and standard deviation of a dataset — are exactly the "Measures of Dispersion" chapter in the NCERT Class 11 Statistics syllabus and appear routinely in JEE Main and BITSAT's statistics questions; the quartile-fence calculation done above is precisely that syllabus's IQR method, so practising it here doubles as board and competitive-exam preparation. The scaling derivation in Stage 3 is standard expectation-and-variance algebra (linearity of E[·], and Var(aX+b) = a²Var(X)) from the Class 11-12 Probability unit, applied to a machine-learning use rather than a textbook random variable — the same algebra, a genuinely new application. The pipeline-construction content in Stages 3-4 and the final section is squarely CBSE's Artificial Intelligence curriculum's Data Literacy and AI Project Cycle units, where "data preprocessing" and "data exploration" are named, assessed skills.

Summary

• A preprocessing pipeline runs four stages in order: handle missing values, detect/handle outliers, scale numeric features, encode categorical features — each with statistics learned only from the training fold and reused, unrecomputed, on the test fold.
• Mean imputation is pulled toward outliers; median imputation resists them — prefer the median whenever a column is skewed or contains extreme values.
• IQR outlier detection: sort the data, find Q1 and Q3 as medians of the lower/upper halves (excluding the overall median when n is odd), then flag anything outside [Q1−1.5·IQR, Q3+1.5·IQR].
• Min-max normalization forces values into [0, 1] via (X−min)/(max−min); z-score standardization forces mean 0 and variance 1 via (X−μ)/σ, provably so by the linearity of E[·] and the a² scaling of Var(·) — and this is what stops large-magnitude features like income from silently dominating small-magnitude features like age in distance-based models.
• Label-encoding a nominal category invents a false order and false distances between categories; one-hot encoding avoids this by giving each category its own independent binary column, at the cost of extra dimensions.
• The single rule holding the whole pipeline together: split first, fit_transform only on the training fold, transform only everywhere else — anything else leaks test information into training.

Practice: Active Recall

Q1. A shop logs 8 transaction amounts (₹): 200, 250, 220, 240, 210, 260, 230, 15000. Compute the mean and the median by hand. Which one would you trust as a "typical transaction size," and why?

Q2. Five students score 45, 60, 55, 70, 50 out of 100 on a test. Using min-max normalization, what is the normalized value of the score 70? Using z-score standardization (population σ), what is the standardized value of 70? (Compute μ and σ from all five scores first.)

Q3. A dataset has a City column with values Chennai, Delhi, Mumbai. A classmate label-encodes it as Chennai=0, Delhi=1, Mumbai=2 and feeds it into a KNN model. Explain concretely, using distances, what false assumption this introduces, and how one-hot encoding avoids it.

Q4. A dataset has 9 values: 4, 7, 8, 9, 10, 12, 13, 15, 40. Using the exclusive-median IQR convention from this chapter (n is odd, so the median is excluded from both halves), find Q1, Q3, IQR, and the upper fence. Is 40 an outlier?

Answer Key

A1. Sum = 200+250+220+240+210+260+230+15000 = 16,610. n = 8. Mean = 16,610 / 8 = 2076.25. Sorted: 200, 210, 220, 230, 240, 250, 260, 15000 — median = average of 4th and 5th values = (230+240)/2 = 235. The median (₹235) is the trustworthy "typical" value; the mean (₹2076.25) is inflated almost tenfold by the single ₹15,000 transaction, exactly the mean-vs-median distortion from Stage 1.

A2. μ = (45+60+55+70+50)/5 = 280/5 = 56. Min-max: (70−45)/(70−45) = 25/25 = 1.0 (70 is the maximum, so it maps to exactly 1). For z-score: variance = [(45−56)²+(60−56)²+(55−56)²+(70−56)²+(50−56)²]/5 = [121+16+1+196+36]/5 = 370/5 = 74, so σ = √74 ≈ 8.60. z(70) = (70−56)/8.60 ≈ 1.63.

A3. Label encoding makes distance(Chennai, Mumbai) = |0−2| = 2 and distance(Chennai, Delhi) = |0−1| = 1, telling any distance-based model that Mumbai is numerically "twice as far" from Chennai as Delhi is — a relationship the city names never actually had; it also implies Delhi is "between" Chennai and Mumbai on some scale, which is meaningless for a nominal category. One-hot encoding replaces the single column with three independent binary columns (is_Chennai, is_Delhi, is_Mumbai), so every pair of distinct cities is equidistant from every other pair, with no invented ordering or magnitude.

A4. Sorted (already sorted): 4, 7, 8, 9, 10, 12, 13, 15, 40. n = 9 is odd, so the median (the 5th value, 10) is excluded from both halves. Lower half = (4, 7, 8, 9) → Q1 = median = (7+8)/2 = 7.5. Upper half = (12, 13, 15, 40) → Q3 = median = (13+15)/2 = 14. IQR = 14 − 7.5 = 6.5. Upper fence = Q3 + 1.5·IQR = 14 + 1.5(6.5) = 14 + 9.75 = 23.75. Since 40 > 23.75, yes, 40 is an outlier.

Think About It

Think about this: How would you explain building a complete data preprocessing pipeline to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.