Cross-Validation and Model Selection

Aisha, Rohan, and Meera are working on the same tiny project: a model that guesses whether a Class 10 student will pass a unit test, using only one number — how many hours they revised the night before. They all use the same six rows of data, the same algorithm (1-nearest-neighbour, or 1-NN: to classify a new student, find the student in the training data whose revision hours are closest, and copy that student's result), and the same rule — hold out two students to test on, train on the other four. Here is the shared dataset, sorted by hours revised:

Aisha holds out the two highest-hours students (9 and 11) to test on. Rohan holds out the two middle students (5 and 7). Meera holds out the two lowest (1 and 3). Same data, same algorithm, same procedure — three different report cards: Aisha gets 100% accuracy, Rohan gets 50%, and Meera gets a flat 0%. Nobody made an arithmetic mistake. Each of them can defend their number. So which one of them actually knows how good this model is?

The honest answer is: none of them, alone — and this chapter is about the fix. A single train-test split is a single roll of the dice; which few points happen to land in the test set can swing your reported accuracy wildly, especially on small data. Cross-validation is the systematic version of what the three of them stumbled into by accident: run every possible split (or a representative set of them), and report the whole distribution of results, not just one lucky or unlucky number.

Why Meera got exactly 0%, and Aisha got exactly 100%

It is worth tracing this by hand once, because the reasoning is the entire chapter in miniature. When Meera removes the two lowest-hours students (1 and 3) from the data, the four students left to train on — 5, 7, 9, 11 — are all "Pass". The 1-NN rule has no "Fail" example left anywhere to point to. Whatever it is asked to predict, it predicts "Pass". So it predicts "Pass" for the student who revised 1 hour (nearest neighbour: 5, distance 4) and "Pass" for the student who revised 3 hours (nearest neighbour: 5, distance 2) — both wrong, because both true labels are "Fail". Zero out of two. This is not the model "being bad at its job"; it is the model being denied any evidence that failing is even possible.

Aisha's split has the opposite problem working in her favour. Training data: 1, 3, 5, 7 (Fail, Fail, Pass, Pass). She asks it to predict for 9 (nearest neighbour: 7, distance 2, label Pass — correct) and for 11 (nearest neighbour: 7, distance 4, label Pass — correct). Two out of two, because both test points sit comfortably on the "obviously going to pass" side, far from the Pass/Fail boundary, so 1-NN cannot get confused.

Rohan's split is the interesting one, because it is the only one that actually tests the boundary. Training data: 1, 3, 9, 11 (Fail, Fail, Pass, Pass). Predicting for 5: distances are 4, 2, 4, 6 to 1, 3, 9, 11 — nearest is 3 (distance 2, label Fail) — but the true label for 5 hours is Pass. Wrong. Predicting for 7: distances are 6, 4, 2, 4 — nearest is 9 (distance 2, label Pass) — true label is Pass. Correct. One out of two: 50%.

Three genuinely different numbers, from three genuinely correct hand-computations, on the same six rows. That is the entire problem cross-validation exists to solve.

k-fold cross-validation, formally

Give the "take turns being the test set" idea a name and a procedure, and you have k-fold cross-validation. Split the dataset into k roughly equal, non-overlapping chunks called folds. Then run k rounds: in round i, fold i is held out as the test set and the remaining k−1 folds are combined to train on. Record the score each round. The final reported number is usually the mean of the k scores, but — and this is the point the three classmates just discovered — the spread across those k scores is itself important information, not noise to be thrown away.

Aisha, Rohan, and Meera's three splits, taken together, are exactly 3-fold cross-validation on this dataset: fold 1 is {9, 11}, fold 2 is {5, 7}, fold 3 is {1, 3}, and each fold takes its turn as the test set while the other two train the model. Nobody needs to guess which of the three splits is "the real answer" — k-fold CV's answer is: report all three, then average them.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import KFold, cross_val_score

# rows ordered so KFold's default contiguous split reproduces
# Aisha's fold, then Rohan's fold, then Meera's fold
X = np.array([[9], [11], [5], [7], [1], [3]])
y = np.array([ 1,   1,   1,   1,   0,   0])   # 1 = Pass, 0 = Fail

knn = KNeighborsClassifier(n_neighbors=1)
kf = KFold(n_splits=3, shuffle=False)

scores = cross_val_score(knn, X, y, cv=kf)
print(scores)              # [1.  0.5 0. ]
print(scores.mean())       # 0.5
print(scores.std())        # 0.408248290463863

Trace it: Python's enumerate counts from 0, so the code calls these rounds 0, 1, 2 — we'll keep calling them Round 1, Round 2, Round 3 in the diagrams and prose below. Round index 0 holds out rows 0–1 (hours 9, 11) — Aisha's split, accuracy 1.0. Round index 1 holds out rows 2–3 (hours 5, 7) — Rohan's split, accuracy 0.5. Round index 2 holds out rows 4–5 (hours 1, 3) — Meera's split, accuracy 0.0. The printed array is exactly the three classmates' report cards, in order. The mean, 0.5, is a far more honest single-number summary of this model than any one of 100%, 50%, or 0% alone — and the standard deviation, about 0.41, tells you something the mean hides completely: this estimate is extremely unstable. A standard deviation almost as large as the mean itself is a loud signal that six data points is nowhere near enough to trust any accuracy number from this model, on this problem, at all.

Why does averaging across folds actually help?

Cricket gives the cleanest intuition. Judging a batter's ability from one innings is a bad idea — a single low score might mean a genuinely weak batter, or it might mean a brilliant yorker on a bad day. Judging them from ten innings and averaging is much better, because the "bad day" and "good day" luck tends to cancel out across a large enough sample, leaving something closer to their true underlying skill. Each cross-validation fold is one innings for your model: a single fold's accuracy mixes real model quality with the luck of which six points happened to end up in that particular test set. Averaging over several folds cancels out some of that luck, the same way averaging over several innings cancels out some of that day-to-day noise.

"Some of," not "all of" — and the reason why is worth being precise about, because it is where most treatments of this topic quietly wave their hands. If you flip ten independent coins and average the results, your average gets more reliable the more coins you add, without any floor on how much better it can get. But cross-validation folds are not independent coins: every pair of folds' training sets overlaps — with 3-fold CV, Aisha's training set and Rohan's training set share half their rows. When two things are built from mostly the same underlying data, their errors tend to move together to some degree: if the model is fooled by an unusual student who appears in most training sets, it tends to be fooled the same way across several folds, not independently each time. This "tendency to move together" is called correlation, and its unnormalised version — how much two quantities vary together, in the same units as their product — is called covariance.

The box below goes past the CBSE Class 10 statistics syllabus, which covers the mean, variance, and standard deviation of a single dataset but not the variance of a sum of two related quantities. It is included because it is the actual mathematical reason folds do not average away as cleanly as independent samples would — a motivated reader who wants to know exactly how much cross-validation buys you should see it once, even a few years before it is formally taught.

Beyond the syllabus — the algebra behind "some of, not all of." Let each fold's error have variance σ², and suppose every pair of folds' errors shares the same covariance, written ρσ² (ρ, the correlation, sits between 0 for "independent" and 1 for "identical"). For the mean of m folds:

Var(mean) = Var( (1/m) Σ e_i )
          = (1/m²) [ Σ_i Var(e_i)  +  Σ_{i≠j} Cov(e_i, e_j) ]
          = (1/m²) [ mσ²  +  m(m−1)ρσ² ]
          = σ²/m  +  ((m−1)/m) ρσ²

Two sanity checks confirm this is doing what it should. If ρ = 0 (folds genuinely independent), the second term vanishes and you get the familiar σ²/m — more folds always help, without limit, exactly like independent coin flips. If ρ is fixed and positive and you let m grow very large, the first term shrinks to zero but the second term approaches ρσ² and stops there — there is a floor on how much averaging can help, set entirely by how correlated the folds are. This is the honest answer to "why not just use 1000 folds and get a perfect estimate": because the folds are not independent, more of them helps less and less, and the return on extra folds bends the wrong way while the compute cost does not.

Stratified k-fold: when a fold can accidentally erase a class

K-fold CV has a specific failure mode that plain averaging cannot fix by itself, and it shows up whenever one class is rare. Suppose a school records, for 12 students, whether they scored a distinction (90%+) in an exam. Only 3 of the 12 did, and — because the roll list happens to be sorted by admission date, and these three joined in the same admission cycle — they sit consecutively at positions 3, 4, and 5:

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold

y = np.array([0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0])   # 1 = distinction, 3 of 12
X = np.arange(12).reshape(-1, 1)

kf_plain = KFold(n_splits=3, shuffle=False)
for i, (tr, te) in enumerate(kf_plain.split(X, y)):
    print("fold", i, "test labels:", y[te], "distinctions in test:", y[te].sum())

Trace it directly: fold 0's test rows are indices 0–3, labels [0, 0, 1, 1] — 2 distinctions. Fold 1's test rows are indices 4–7, labels [1, 0, 0, 0] — 1 distinction. Fold 2's test rows are indices 8–11, labels [0, 0, 0, 0] — zero distinctions. In round 2, the model is evaluated on a test set that contains not a single example of the class you presumably care most about measuring — you cannot compute "how often does it correctly spot a distinction student" when there are no distinction students to spot. Worse, in round 0, the training set for that round only has 1 of the 3 distinction examples to learn from, since the other 2 were removed with the test fold.

StratifiedKFold fixes this by building each fold to preserve the overall class proportions, rather than just taking contiguous chunks of whatever order the data happens to be in. With exactly 3 distinction students and 3 folds, the arithmetic divides evenly, so stratification can guarantee — not just aim for — exactly 1 distinction student in every fold's test set:

skf = StratifiedKFold(n_splits=3, shuffle=False)
for i, (tr, te) in enumerate(skf.split(X, y)):     # same 12-row X, y as above
    print("fold", i, "distinctions in test:", y[te].sum())
# fold 0 distinctions in test: 1
# fold 1 distinctions in test: 1
# fold 2 distinctions in test: 1

It is worth flagging the limit of this fix honestly, because pretending it is a complete cure would be its own misconception: stratification can only distribute what exists. If there had been only 2 distinction students and you still asked for 3 folds, no algorithm could put a whole student's worth of distinction-label into all three folds — at least one fold would still get zero, because you cannot split 2 indivisible examples three ways without a remainder. The real rule of thumb is that your fold count should never exceed your rarest class's count; if it does, reduce k before reaching for stratification.

Common misconception: "cross-validation trains the model"

It does not, and treating it as if it does is one of the most common errors in early ML work. Cross-validation's entire job is to estimate how well a modelling approach will generalise — it produces a number (or a small set of numbers, one per fold), not a deployable model. Each of the k rounds trains and then throws away its own temporary model on k−1 folds. None of those k models is "the" model. Once cross-validation has told you which algorithm or which hyperparameter setting looks best, the standard next step is to retrain fresh, one final time, on all of the available data (all six of the classmates' rows, not just four of them) using that chosen setting — because more training data is always better once you are done choosing, and none of the k intermediate models used all of it.

A second mix-up worth naming explicitly, because the vocabulary invites it: the k in "k-fold cross-validation" and the k in "k-nearest-neighbours" are two completely unrelated numbers that happen to share a letter by historical accident. KFold(n_splits=3) and KNeighborsClassifier(n_neighbors=1) are answering different questions — "how many test rounds should I run?" versus "how many nearby training points should vote?" — and setting one has no bearing on the other. Reading code that mixes both, like the snippet above, is exactly where students conflate them.

Model selection: using cross-validation to choose, not just to measure

Once you trust the mean-of-k-folds number more than any single split, the natural next move is to compute it for several candidate models and pick the one with the best mean. Try 1-NN (n_neighbors=1) against 3-NN (n_neighbors=3) on the same six rows, same three folds:

from sklearn.model_selection import GridSearchCV

# back to the six-row revision-hours dataset, not the 12-row one above
X = np.array([[9], [11], [5], [7], [1], [3]])
y = np.array([ 1,   1,   1,   1,   0,   0])
kf = KFold(n_splits=3, shuffle=False)

knn = KNeighborsClassifier()
grid = GridSearchCV(knn, param_grid={"n_neighbors": [1, 3]}, cv=kf)
grid.fit(X, y)
print(grid.best_params_)

Hand-tracing 3-NN on the same three rounds (Round 1 = Aisha's split, Round 2 = Rohan's, Round 3 = Meera's), remembering that with only four training points each round, "3 nearest" simply means "leave out the single farthest one": Round 1 trains on hours 5, 7, 1, 3 (labels Pass, Pass, Fail, Fail). Predicting for 9: distances to {5, 7, 1, 3} are 4, 2, 8, 6 — the farthest is 1, so the three nearest are 7, 5, 3 — votes Pass, Pass, Fail — majority Pass, correct. Predicting for 11: distances are 6, 4, 10, 8 — farthest is again 1, so the three nearest are again 7, 5, 3 — majority Pass, correct. Round 1 is 2/2. Round 2 trains on hours 9, 11, 1, 3 (labels Pass, Pass, Fail, Fail). Predicting for 5: distances are 4, 6, 4, 2 — farthest is 11, so the three nearest are 3, 9, 1 — votes Fail, Pass, Fail — majority Fail, but the true label is Pass — wrong. Predicting for 7: distances are 2, 4, 6, 4 — farthest is 1, so the three nearest are 9, 11, 3 — votes Pass, Pass, Fail — majority Pass, correct. Round 2 is 1/2. Round 3 trains on hours 9, 11, 5, 7, all four Pass, so every 3-NN vote is unanimously Pass regardless of which three are picked — both test points (1 and 3, both true Fail) are misclassified: 0/2. So 3-NN's three fold scores are also [1.0, 0.5, 0.0], mean 0.5 — identical to 1-NN.

This tie is itself the lesson, not an inconvenience to skip past. With only six data points split into three folds, each training fold has just four rows — there simply is not enough data here for the choice between k=1 and k=3 to show a difference. GridSearchCV still has to report something as the winner, and its tie-breaking rule picks whichever candidate it evaluated first among those sharing the best score — here, n_neighbors=1, since it is listed first in the grid. That result should not be read as "1-NN is better"; it should be read as "this dataset is too small to tell them apart, and a tool that must output a single best answer will sometimes output one that isn't meaningfully better than the alternative." Comparing against a floor is more informative here than comparing 1-NN to 3-NN: take a baseline that ignores the hours feature entirely and always predicts whichever class is more common in its training fold (ties broken toward Fail, the lower-coded class). Round 1's training labels are Pass, Pass, Fail, Fail — a tied 2-2 majority, so the baseline predicts Fail for both test points, hours 9 and 11, whose true label is Pass — both wrong, 0/2. Round 2's training labels, for hours 9, 11, 1, 3, are Pass, Pass, Fail, Fail again — another 2-2 tie, baseline predicts Fail for both test points, hours 5 and 7, true label Pass — both wrong, 0/2. Round 3's training labels are unanimously Pass, so the baseline predicts Pass for both test points, hours 1 and 3, true label Fail — both wrong, 0/2. The baseline scores 0/2 in every round, mean 0.0, against 1-NN and 3-NN's mean of 0.5 — confirming that whatever the neighbour-based models are doing, it is capturing real signal from the hours feature, not just guessing the majority class.

How many folds, and the special case of leave-one-out

The choice of k trades off two things directly, because of the very algebra derived above. A larger k means each round trains on more data (only 1/k of the rows are held out), which tends to reduce bias — the estimate is closer to what you would get training on the full dataset. But a larger k also means the training sets across rounds overlap more (with k=10, any two folds' training sets share roughly 8/9 of their rows), which pushes ρ upward and, by the formula above, raises the variance floor of the averaged estimate. This is the bias-variance trade-off applied to the cross-validation estimate itself, not to the model being evaluated.

Pushed to its extreme, k equal to the number of data points is called leave-one-out cross-validation (LOOCV): every round trains on all but one point and tests on that single point. It uses the maximum possible amount of training data each round (lowest bias) but is the most expensive to compute (one full retrain per data point — infeasible once you have thousands of rows) and, being built from the most-overlapping training sets possible, tends to have the highest correlation ρ between rounds. In practice, k = 5 or k = 10 is the standard default across most machine learning work, chosen empirically as a reasonable middle ground rather than derived from a single formula — you will see cv=5 as scikit-learn's own default for exactly this reason.

The leakage trap: fit your preprocessing only on the training fold

The single most common way to silently inflate a cross-validation score is to compute something from the entire dataset before splitting into folds — the mean and standard deviation for feature scaling, for instance, or which words appear frequently enough to keep as features in a text model. If you calculate a feature's mean using all 12 students and then use that mean to scale both the training and test rows of every fold, each "test" fold has quietly leaked a little information about itself into the numbers the model was allowed to see, because the model's scaling now reflects the very rows it is about to be scored on. The score you get back is a measurement of a model that has partially memorised its own answer key, and it will not hold up on genuinely new students. The fix is procedural discipline: any computation that looks at labels or feature statistics — scaling, encoding rare categories, selecting top features by correlation with the target — must be fit only on each fold's training rows, then applied unchanged to that fold's test rows. Scikit-learn's Pipeline combined with cross_val_score enforces exactly this automatically, which is the main reason to prefer it over manually scaling the whole dataset before calling KFold.

Where this fits in your CBSE work

CBSE's Artificial Intelligence curriculum for the senior years treats model evaluation as a hands-on practical skill rather than a formula to memorise for an objective-type question: you are expected to load a small dataset, split it, train a model with a library such as scikit-learn, and evaluate it as part of project and practical-file work, with viva questions that probe whether you understand why a single split can mislead rather than just how to type the code. That is precisely the reasoning this chapter walked through by hand with the six-student dataset — if you can independently trace why Meera's split gave 0% while Aisha's gave 100%, and explain in your own words why StratifiedKFold matters for the 12-student example, you are prepared for exactly the kind of question a practical exam or project viva asks, not just the code that produces a number.

Check yourself

• Using the six-row dataset, if instead the two test rows in a round were hours 3 and 9 (train on 1, 5, 7, 11): for hours 9, the nearest neighbours by distance are 7 and 11, both distance 2, and both labelled Pass, so the tie doesn't change the prediction. But for hours 3, the nearest neighbours are 1 (distance 2, Fail) and 5 (distance 2, Pass) — a genuine tie between two different labels. Trace both by hand, then answer: since real 1-NN implementations must return a single prediction even on an exact tie, what does this particular tie tell you about how much you should trust 1-NN's output near a decision boundary on a dataset this small and evenly spaced?
• Explain in one sentence why cross_val_score returning [1.0, 0.5, 0.0] is more useful to report than just its mean, 0.5.
• A dataset has 200 rows, of which only 4 belong to the rare class. Why would StratifiedKFold(n_splits=10) still leave some folds with zero rare-class examples in the test set, and what is the more appropriate fix?
• A classmate scales their features using X.mean() computed once over the full dataset, then runs 5-fold cross-validation. Why is their reported accuracy likely to be optimistic, and what should they do instead?
• If two folds' errors have correlation ρ = 0.6 and each has variance σ² = 0.04, use the derived formula to find the variance of the mean over m = 5 folds, and compare it to what the variance would have been if the folds were independent (ρ = 0).

Summary

A single train-test split reports one point on a distribution of possible outcomes, and on small data that distribution can be wide enough to make 0%, 50%, and 100% all individually "correct" answers to the same question, as the three classmates discovered on the same six rows. K-fold cross-validation replaces that one lucky-or-unlucky number with the mean (and, just as importantly, the spread) across k rounds, each holding out a different fold as the test set. Averaging helps because the rounds' errors are not perfectly correlated, but it does not help without limit, because they are not independent either — the folds share training data, and that shared data sets a floor on how much averaging can shrink your uncertainty, governed by the covariance between folds. Stratified k-fold prevents plain k-fold's specific failure of accidentally emptying a fold of a rare class, though it cannot manufacture examples that do not exist. Cross-validation's output is a decision-making tool for choosing between models or hyperparameters — not a trained model itself — and the model you actually ship should be retrained on the full dataset once that choice is made. Get the number honestly, and the last mistake to avoid is leaking information about the test fold into how you preprocessed the training fold in the first place.

Think About It

Think about this: How would you explain cross-validation and model selection to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.