Cross-Validation and Model Selection: Rigorous ML Evaluation

The 98% Model That Failed Every New Student

Imagine you are entering a school science exhibition with an AI project: given a student's weekly study hours, attendance percentage, and mock-test score, predict whether they will clear the CBSE Class 10 boards. You collect data from 50 students in your school, train a decision tree on all 50, and then check how many of those same 50 students it classifies correctly. The answer comes back: 49 out of 50 — 98% accuracy. You are ready to present it as a triumph.

Then a judge asks one question that deflates the whole project: "You tested it on the same students it learned from. How do you know it works on a student it has never seen?" You have no answer, because you never checked. The tree may have simply memorised that "Rohan studied 14 hours and passed" rather than learning any real relationship between study habits and results. A model that has memorised its training data can score close to 100% on that data while being nearly useless on anyone new — this gap between performance on seen data and performance on unseen data is exactly what overfitting means, and training accuracy alone can never detect it, because the model is being graded on questions it already knows the answers to.

The fix seems obvious: hold back some students, train only on the rest, and test only on the held-back group. This is a train/test split, and it is a genuine improvement — but by itself it is not enough, and understanding exactly why is the real starting point of this chapter.

Why One Split Isn't Enough — The Variance Problem

Suppose your 50 students are split 80/20: 40 for training, 10 held out for testing. You run this once and get 90% test accuracy. Curious, you reshuffle and try a different random 10 students as the test set. This time you get 70%. Reshuffle again — 85%. Nothing about your model changed between these three runs; only which 10 students happened to land in the test set changed. With only 10 test students, a handful of unusually easy or unusually hard cases can swing the reported accuracy by 20 percentage points purely by chance.

This is not a hypothetical worry — it is a direct consequence of small-sample statistics. A test accuracy computed on n test points is itself a random variable with its own variance, and that variance shrinks only as you test on more points. A single 20% holdout on 50 students gives you just 10 data points to estimate performance from, which is far too few to trust. You need a way to test on every student while still never testing a model on data it was trained on. That is precisely the problem cross-validation solves.

k-Fold Cross-Validation, Defined Precisely

Take your dataset of n examples and randomly partition it into k equal-sized groups called folds, labelled F₁, F₂, …, F_k. Now run k separate experiments. In experiment j, hold out fold F_j as the validation set, train the model on the union of all the other k − 1 folds, and compute a score e_j (accuracy, error rate, or whatever metric you care about) by evaluating that trained model on F_j alone. After k experiments, every single example has been used for validation exactly once — and, crucially, a model is never evaluated on data it was trained on.

The final cross-validation score is the average of the k per-fold scores:

CV_score = (1/k) * (e_1 + e_2 + ... + e_k)

The diagram below shows this for k = 5 on a dataset split into five folds. Each row is one experiment; the amber block is that experiment's validation fold, the blue blocks are what the model trains on. Notice that across the five rows, every fold takes a turn as the amber block exactly once — that single-turn guarantee is the defining property of k-fold CV.

Read off the five validation scores from the diagram: 0.90, 0.85, 0.80, 0.88, 0.82. Their mean is (0.90+0.85+0.80+0.88+0.82)/5 = 4.25/5 = 0.85. To know how much this mean would jump around if we had drawn a different random partition into folds, compute the sample standard deviation of the five scores. The deviations from the mean are +0.05, 0, −0.05, +0.03, −0.03; squaring and summing gives 0.0025+0+0.0025+0.0009+0.0009 = 0.0068; dividing by (k − 1) = 4 gives a sample variance of 0.0017; the square root gives a sample standard deviation of about 0.0412. The standard error of the mean is this standard deviation divided by √k:

SE = 0.0412 / sqrt(5) = 0.0412 / 2.236 ≈ 0.0184

So the honest way to report this result is "85% ± 1.8% accuracy," not a bare "85%." A model reporting 85% ± 1.8% and a model reporting 87% ± 4% are not distinguishably different — this SE is exactly what lets you make that judgement instead of chasing noise. One caveat worth knowing at this level: this SE formula assumes the k fold-errors behave like independent samples, but they are not fully independent — each pair of folds shares almost all of its training data with the others, so this SE understates the true uncertainty somewhat. Treat it as a useful, standard approximation, not an exact statistical guarantee.

Worked Example — Leave-One-Out Cross-Validation by Hand

The extreme case of k-fold CV is k = n: every single example gets its own fold, so each round trains on all n − 1 other points and validates on the one left out. This is called Leave-One-Out Cross-Validation (LOOCV). Let's trace it completely by hand, so the mechanism is fully transparent rather than a black box.

Take six students with hours studied per week (X) and Pass(1)/Fail(0) outcome:

A: X=1, Fail(0)      D: X=7, Pass(1)
B: X=2, Fail(0)      E: X=8, Fail(0)  <- an outlier: studied a lot, still failed
C: X=3, Fail(0)      F: X=9, Pass(1)

Use a 1-nearest-neighbour classifier: to predict a left-out point, find the single closest remaining point (by |difference in hours|) and copy its label. Trace all six rounds:

• Omit A(1): nearest remaining is B(2), distance 1, label Fail. True label of A is Fail. Correct.
• Omit B(2): A(1) and C(3) are tied at distance 1 — but both are labelled Fail, so the prediction is Fail regardless of the tie. True label of B is Fail. Correct.
• Omit C(3): nearest remaining is B(2), distance 1, label Fail. True label of C is Fail. Correct.
• Omit D(7): nearest remaining is E(8), distance 1, label Fail. True label of D is Pass. Incorrect — the outlier E pulled D's prediction down.
• Omit E(8): D(7) and F(9) are tied at distance 1, both labelled Pass, so prediction is Pass. True label of E is Fail. Incorrect — E is the noisy point itself; no neighbour could have predicted it correctly.
• Omit F(9): nearest remaining is E(8), distance 1, label Fail. True label of F is Pass. Incorrect — again dragged down by E.

Three correct, three incorrect: LOOCV accuracy = 3/6 = 50%. Compare this to training accuracy: a 1-nearest-neighbour model evaluated on its own training set is always 100% accurate, because every point's nearest neighbour — when it is allowed to see itself — is itself, at distance zero. This is precisely the "98% model" trap from the opening story, reproduced in miniature: training accuracy is trivially perfect and tells you nothing, while LOOCV exposes that a single noisy label (E) contaminates every prediction near it, cutting true accuracy in half. You can reproduce this exact result in code:

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import LeaveOneOut, cross_val_score

X = np.array([[1], [2], [3], [7], [8], [9]])   # hours studied
y = np.array([0, 0, 0, 1, 0, 1])                # Fail=0, Pass=1; E=8 is the outlier

knn1 = KNeighborsClassifier(n_neighbors=1)
scores = cross_val_score(knn1, X, y, cv=LeaveOneOut())

print(scores)        # [1. 1. 1. 0. 0. 0.]  (order follows A,B,C,D,E,F)
print(scores.mean()) # 0.5

How Many Folds? The Bias-Variance Tradeoff in k

Why not always use LOOCV, since it uses the most training data possible in every round? Because of a genuine tradeoff. As k grows toward n:

• Bias falls: each training set has n − n/k examples, so a larger k means each round trains on more data, closer to what you'd get training on the full dataset — the score is a less biased estimate of "accuracy if trained on everything."
• Variance of the estimate can rise: with LOOCV, the n training sets overlap in n − 2 out of n − 1 points with each other — they are nearly identical. Averaging n highly correlated, nearly-identical experiments does not cancel out noise the way averaging n independent experiments would, so the final LOOCV score can swing more than a k-fold score would if you reran the whole procedure on a fresh sample.
• Compute cost rises: LOOCV requires training the model n times; for n = 10,000 that is 10,000 full training runs.

This is why k = 5 or k = 10 is the standard default in practice: enough folds that each training set is close to the full data (low bias), but few enough that the folds are meaningfully different from each other (kept variance under control) and the computation stays affordable.

Stratified k-Fold — When Classes Aren't Balanced

Plain random k-fold splitting can go badly wrong when one class is rare. Consider a model flagging suspicious UPI transactions as fraudulent — as in most real fraud-detection settings, genuine transactions vastly outnumber fraudulent ones. Suppose out of 1,000 transactions, only 20 are fraud (2%). With plain random 5-fold splitting, pure chance could place 8 of those 20 fraud cases in one fold and only 1 in another. A fold with almost no fraud examples in training makes the model unable to learn what fraud looks like for that round, and a validation fold with almost no fraud examples makes the fraud-detection score for that round nearly meaningless — you could score 98% accuracy just by labelling everything "genuine."

Stratified k-fold fixes this by forcing each fold to preserve the overall class ratio: if 2% of the whole dataset is fraud, each of the 5 folds is built to contain roughly 2% fraud too. This makes every round a fair, representative test rather than a lottery on how the rare class happened to scatter. In scikit-learn this is `StratifiedKFold` instead of plain `KFold`, and for classification tasks with any meaningful class imbalance it should be your default, not an afterthought.

Using Cross-Validation for Model Selection

Cross-validation's second major use — beyond honestly estimating a fixed model's accuracy — is choosing between models or hyperparameters. Return to the K in K-nearest-neighbours. On our six-point toy dataset, try K = 3 instead of K = 1: leave-one-out still gives exactly 3 correct out of 6 (A, B, C correct; D, E, F still wrong — the outlier E is close enough to drag down its neighbours' 3-nearest-neighbour votes too). This is an important, honest result in itself: with only six points and one genuinely noisy label, no choice of K can rescue the outlier's neighbours, because there simply isn't enough data for the noise to average out. This is exactly why real K-selection curves are built from datasets of hundreds or thousands of points, not six.

On a realistically-sized dataset, running k-fold CV separately for each candidate K produces a pattern like the illustrative one below (typical shape, not real study data):

This is the classic U-shape: very small K overfits (memorises local noise, like our K=1 example), very large K underfits (averages over so many neighbours that real structure gets smoothed away), and there is a sweet spot in between. Simply picking the single highest mean (K = 7, at 0.89) is tempting but often unwise, because 0.89 ± 0.02 and 0.885 ± 0.02 are not meaningfully different — the gap is smaller than the noise in the estimate. The one-standard-error rule (from Hastie, Tibshirani & Friedman's Elements of Statistical Learning) handles this properly: find the best mean score (here 0.89), subtract one SE to get a tolerance band (0.89 − 0.02 = 0.87), and among all models whose mean score clears that band, pick the simplest one — in KNN, "simpler" means larger K, since more neighbours means a smoother, less flexible, more heavily regularised decision boundary. K = 5, 7, and 9 all clear 0.87, so the rule selects K = 9 over the raw-best K = 7: statistically indistinguishable performance, but a simpler, more stable model that is less likely to have gotten lucky.

The Cardinal Sin — Data Leakage During Cross-Validation

Misconception: "As long as I'm doing cross-validation, my evaluation is automatically fair." Correction: cross-validation only protects you if every step that touches the labels or looks at the full dataset is redone independently inside each fold. The single most common way students break this is preprocessing before splitting.

Suppose you min-max scale your "hours studied" feature to the range [0, 1] using the minimum and maximum of the entire dataset, and only afterward run 5-fold CV. Each training fold's scaler now "knows" the maximum value from students who are sitting in that round's validation fold — information the model should not have had access to during training. If the global maximum happens to be 20 hours but a particular training fold on its own only goes up to 15 hours, scaling by the global 20 gives systematically different, leaked-information numbers than scaling by that fold's own 15 would. The fix is to fit every preprocessing step — scaling, feature selection, PCA, imputation — only on each round's training fold, and merely apply (never refit) that same transform to the validation fold. In scikit-learn, wrapping the scaler and the model together in a `Pipeline` and passing the whole pipeline to `cross_val_score` does this automatically and correctly.

Nested Cross-Validation — Tuning and Evaluating Without Cheating

A second, subtler leakage happens even when preprocessing is done correctly. If you use one round of 5-fold CV to try K = 1, 3, 5, 7, 9 and then report "our best CV accuracy was 0.89" as your model's final performance, that number is optimistic — you effectively searched five options and reported whichever one got lucky with the folds, so you've indirectly let the validation folds influence which model you present. The fix is nested cross-validation:

1. Split the data into an outer set of k folds (say 5).
2. For each outer fold, hold it out as the outer-test set, and on the remaining outer-training data, run a complete inner cross-validation (say another 5-fold) to search over candidate hyperparameters and pick whichever performs best on the inner CV.
3. Retrain a model using that chosen hyperparameter on the full outer-training data, and evaluate it exactly once on the outer-test fold — data the hyperparameter search never touched.
4. Average the 5 outer-test scores. This average is an honest estimate of how well "the whole procedure, including its own hyperparameter search" will generalise — not the score of one cherry-picked hyperparameter.

In scikit-learn, nested CV is simply `GridSearchCV` (the inner loop) passed as the estimator into an outer `cross_val_score`.

from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn.neighbors import KNeighborsClassifier

inner_cv = KFold(n_splits=5, shuffle=True, random_state=1)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=2)

param_grid = {"n_neighbors": [1, 3, 5, 7, 9]}
search = GridSearchCV(KNeighborsClassifier(), param_grid, cv=inner_cv)

nested_scores = cross_val_score(search, X, y, cv=outer_cv)
print(nested_scores.mean())  # honest generalisation estimate

Two Misconceptions, Corrected

Misconception 1: "Cross-validation prevents overfitting." Correction: cross-validation does not change the model or stop it from overfitting; it only measures how badly a model overfits, giving you an honest number so you can choose a less complex model, add regularisation, or gather more data. CV is a diagnostic tool and a selection tool, not a cure by itself — this chapter's LOOCV example still overfit at K=1 whether or not we measured it; measuring it with LOOCV is what let us notice.

Misconception 2: "A model's cross-validation accuracy is a single fixed truth about the model." Correction: it is an estimate with its own uncertainty, quantified by the standard error computed from the spread of per-fold scores. Two models whose CV scores differ by less than roughly one SE should generally be treated as tied, not ranked — this is exactly the reasoning behind the one-standard-error rule used above.

Where This Fits: CBSE and Competitive Exams

If you're taking CBSE's Artificial Intelligence skill subject (Code 417 in Classes 9–10, and Code 843 as a Class 11–12 elective), model evaluation and generalisation are explicit learning outcomes, and cross-validation is the standard rigorous method behind them — expect exam questions that ask you to distinguish training, validation, and test data, or to explain why a single train/test split can mislead. The standard-error calculation above is direct practice for the sampling-distribution and standard-error questions that appear in Class 11–12 Applied Mathematics statistics, and by extension in JEE/BITSAT-style quantitative sections that test mean, variance, and standard deviation computation under time pressure. And if you ever build a real ML project for a school science exhibition, an Inspire Award submission, or a hackathon, "How did you validate this, and how confident are you in that number?" is close to the first question any serious judge will ask — cross-validation, reported with its standard error, is the rigorous answer.

Summary

A single train/test split gives a noisy, unreliable performance estimate because it wastes most of the data and depends on the luck of one particular split. k-fold cross-validation fixes this by rotating every example through the validation role exactly once and averaging the k scores, with the spread of those scores giving a standard error you should always report alongside the mean. LOOCV (k = n) has the least bias but the most variance and cost; k = 5 or 10 is the practical sweet spot. Stratified k-fold preserves class proportions in every fold and is essential whenever classes are imbalanced. Cross-validation also drives model selection — comparing hyperparameters like K in KNN — where the one-standard-error rule favours the simplest model within noise of the best. Preprocessing must be refit inside each training fold to avoid leakage, and when both tuning and evaluating, nested cross-validation keeps the final reported number honest.

Active Recall

1. A 10-fold CV run gives these accuracies: 0.92, 0.89, 0.94, 0.90, 0.91, 0.93, 0.88, 0.90, 0.92, 0.91. Compute the mean and, without doing the full standard-deviation arithmetic, state whether you'd expect the standard error to be smaller or larger than the 5-fold example in this chapter, and why. (Mean = 9.10/10 = 0.91. More folds close together in value plus a larger k in the √k denominator both push SE down — expect a smaller SE than the 5-fold case.)
2. Explain, in one sentence, why a 1-nearest-neighbour model always scores 100% on its own training set, and why that number is meaningless for judging the model.
3. Spot the leakage: a student computes the mean and standard deviation of a feature across the entire dataset, standardises the feature using those numbers, and only then runs `cross_val_score`. What went wrong, and what is the one-line fix?
4. Two hyperparameter settings score 0.912 ± 0.015 and 0.905 ± 0.014 on 10-fold CV. Should you always pick the first because its mean is higher? Justify using the one-standard-error idea.
5. Why does nested cross-validation need an inner loop and an outer loop instead of just one round of CV over several hyperparameters?