Hyperparameter Tuning: The Art of Model Optimization

How Many Friends Should You Ask?

Suppose you want to decide whether a new movie is worth watching this weekend before you book tickets. You could ask exactly one friend who already watched it — but what if that one friend has unusual taste? You could instead ask five friends and go with whatever the majority says. Or you could ask all fifteen people in your class WhatsApp group and go with the crowd. Which number of friends should you ask?

Notice something important: nobody can calculate the "correct" number of friends to ask using a formula. You have to pick a number, see how well it works, maybe try a different number, and compare. This exact situation — choosing a setting that controls how a prediction system behaves, before you can even measure whether it works — is the central problem of this chapter. In machine learning, that setting is called a hyperparameter, and the process of searching for good values is called hyperparameter tuning.

Parameters vs. Hyperparameters: Two Very Different Kinds of Numbers

Every machine learning model has two categories of numbers inside it, and mixing them up is one of the most common early mistakes.

• Parameters are values the algorithm figures out automatically by looking at the training data. In linear regression, the slope and intercept of the best-fit line are parameters — the algorithm calculates them from the data points, and you never type them in by hand.
• Hyperparameters are values you decide before the algorithm even starts running. They are not learned from data; they control how the algorithm learns or predicts. The algorithm has no way to discover the best hyperparameter on its own — that job belongs to you, the person building the model.

The "how many friends to ask" number is a perfect hyperparameter, and this chapter will use it as a running example through an algorithm called k-Nearest Neighbors (KNN), where that number is literally named k. A few other hyperparameters you will encounter as you go deeper into machine learning, so you can recognize the pattern: in Decision Trees, max_depth controls how many yes/no questions deep the tree is allowed to grow. In k-Means clustering, the number of clusters is chosen before the algorithm runs, not discovered by it. In more advanced systems like neural networks (which you will meet in later grades), the learning rate controls how big a step the model takes while adjusting itself on each round of training. All of these share the same property: nobody can compute their "correct" value from a formula. You must search for good values, exactly like choosing how many friends to ask.

The Dataset We Will Tune Against

To make hyperparameter tuning concrete, we will build a tiny KNN model that predicts whether a student passed or failed a term exam, based on two measurements: hours studied per day, and attendance percentage. Here is our training data — ten students whose actual results we already know:

Six students passed and four failed. Keep that 6-to-4 split in mind — it will explain something surprising later in this chapter. Also notice that the two columns live on very different numeric scales; we will come back to why that matters when we reach the practice questions.

A Quick Refresher: How KNN Actually Predicts

KNN predicts a new student's result by finding the k training students who are numerically "closest" to them, and letting those k students vote by majority. "Closest" is measured using a distance formula between two points. You will meet the formal distance formula in your Class 10 Coordinate Geometry chapter — here is how it works so we can put it to use right now:

For two points (x₁, y₁) and (x₂, y₂), the straight-line (Euclidean) distance between them is:

distance = √[ (x₁ − x₂)² + (y₁ − y₂)² ]

Let's use it. A new student, Rohan, studies 6 hours a day and has 65% attendance — point (6, 65). We want to know: is he more like the students who passed, or the students who failed? Let's hand-calculate his distance to two training students.

Distance to Meera (6, 70): study-hours difference = 6 − 6 = 0. Attendance difference = 65 − 70 = −5. Distance = √(0² + (−5)²) = √25 = 5.00.

Distance to Rahul (5, 60): study-hours difference = 6 − 5 = 1. Attendance difference = 65 − 60 = 5. Distance = √(1² + 5²) = √26 ≈ 5.10.

Doing this same calculation for all ten training students and sorting from nearest to farthest gives:

Now watch what happens as we change k, the hyperparameter:

• k = 1: only Meera votes → Pass.
• k = 3: Meera(P), Rahul(F), Sneha(P) → 2 Pass, 1 Fail → Pass.
• k = 5: add Karan(F), Divya(P) → 3 Pass, 2 Fail → Pass.
• k = 7: add Vikram(F), Priya(P) → 4 Pass, 3 Fail → Pass.
• k = 9: add Anjali(P), Hari(P) → 6 Pass, 3 Fail → Pass.

For Rohan, every value of k agrees: Pass. That will not always happen — and the cases where it does not are exactly where hyperparameter tuning earns its keep.

What Goes Wrong When k Is Too Small or Too Large

Here is a simplified, hypothetical situation (not from our student table) that shows the risk of setting k too small. Imagine five training points near a new query, sorted nearest to farthest, with their known outcomes: 1st-nearest = Fail, 2nd-nearest = Pass, 3rd-nearest = Pass, 4th-nearest = Pass, 5th-nearest = Fail. If we set k = 1, the model trusts only the single nearest point and predicts Fail — built entirely on one neighbor that might simply be an unusual, noisy case (a student who failed for a reason unrelated to study hours or attendance, like illness during the exam). If we set k = 3, the model looks at three neighbors — Fail, Pass, Pass — and predicts Pass by majority, smoothing over that one odd data point. When k is too small, the model becomes overly sensitive to noise in individual data points. This is a form of overfitting: the model reacts to one-off quirks in the training data instead of the real underlying pattern.

The opposite failure happens when k is too large. Recall that our training data has only 4 Fail students out of 10. If k gets close to 10, the model is forced to include almost every training point in its vote — including many that are far away and not really "similar" to the new student at all. At that point, the vote stops reflecting the new student's actual neighborhood and starts reflecting the dataset's overall Pass/Fail split. This is underfitting: the model becomes too simple to capture real local patterns, and just repeats the majority class. We will see this happen with real numbers in the next section.

Training, Validation, and Test: Why You Need Separate Buckets of Data

There is a trap hiding in how we just evaluated Rohan: we cannot judge hyperparameters using the same data the model was built from. Here is why, using our own KNN model as proof. If we set k = 1 and then "tested" the model on the ten training students themselves, it would score a perfect 100% every single time — not because k = 1 is a good hyperparameter, but because each training student's own nearest neighbor, at a distance of exactly 0, is itself. The model would just be reading back labels it already had memorized. That tells us nothing about how it will behave on a real new student.

This is why machine learning practice splits labelled data into separate buckets: a training set the model learns from, a validation set used only to compare different hyperparameter choices, and a test set touched exactly once, at the very end, to report an honest final accuracy. Our ten students form the training set. To tune k fairly, we need a separate group of students whose results are already known but who were not used to build the neighbor list. Here is our validation set of four such students:

For each of these four students, we already showed KNN correctly predicts Pass for Rohan across every tested k. Working through the neighbor lists for Neha and Suresh by hand (using the same distance formula and the same sorted-neighbor method shown above) gives a striking pattern: at k = 7, Neha's seven nearest neighbors split 4 Fail to 3 Pass — correctly predicting Fail. But at k = 9, two more Pass-labelled students get pulled into the vote, flipping it to 5 Pass vs. 4 Fail — an incorrect prediction. The exact same flip happens for Suresh: correct (4 Fail–3 Pass) at k = 7, wrong (5 Pass–4 Fail) at k = 9. This is underfitting in action: with only 4 Fail students in the whole training set, once k reaches 9 out of 10 possible neighbors, the vote can barely avoid echoing the dataset's overall 6-Pass-to-4-Fail majority, regardless of who the new student actually resembles locally.

Grid Search: Trying Every Combination Systematically

Instead of guessing one value of k and hoping, we can search a whole range of candidate values — and, since KNN also needs a choice of distance formula, we can search combinations of two hyperparameters at once: k and the distance metric (Euclidean, which we used above, or Manhattan, which adds up straight-line horizontal and vertical differences instead of using a square root: |x₁ − x₂| + |y₁ − y₂|). Testing every combination of hyperparameter values, laid out like cells in a table, is called grid search. Here is the full, working implementation:

training_data = [
    ((2, 40), "Fail"),   # Amit
    ((8, 85), "Pass"),   # Priya
    ((5, 60), "Fail"),   # Rahul
    ((7, 75), "Pass"),   # Sneha
    ((3, 50), "Fail"),   # Vikram
    ((9, 88), "Pass"),   # Anjali
    ((4, 55), "Fail"),   # Karan
    ((6, 70), "Pass"),   # Meera
    ((8, 80), "Pass"),   # Divya
    ((5, 90), "Pass"),   # Hari
]
# Each point is (StudyHours, Attendance%)

def euclidean(p1, p2):
    return ((p1[0] - p2[0]) ** 2 + (p1[1] - p2[1]) ** 2) ** 0.5

def manhattan(p1, p2):
    return abs(p1[0] - p2[0]) + abs(p1[1] - p2[1])

def knn_predict(query, training_data, k, metric):
    distance_fn = euclidean if metric == "euclidean" else manhattan
    distances = [(distance_fn(query, point), label) for point, label in training_data]
    distances.sort(key=lambda pair: pair[0])
    nearest_k = distances[:k]
    pass_votes = sum(1 for _, label in nearest_k if label == "Pass")
    fail_votes = k - pass_votes
    return "Pass" if pass_votes > fail_votes else "Fail"

validation_data = [
    ((6, 65), "Pass"),   # Rohan
    ((3, 45), "Fail"),   # Neha
    ((7, 82), "Pass"),   # Tanvi
    ((4, 58), "Fail"),   # Suresh
]

def validation_accuracy(k, metric, training_data, validation_data):
    correct = 0
    for query, true_label in validation_data:
        prediction = knn_predict(query, training_data, k, metric)
        if prediction == true_label:
            correct += 1
    return correct / len(validation_data)

best_accuracy = 0
best_params = None
for k in [1, 3, 5, 7, 9]:
    for metric in ["euclidean", "manhattan"]:
        acc = validation_accuracy(k, metric, training_data, validation_data)
        print(f"k={k}, metric={metric}: accuracy={acc:.2f}")
        if acc > best_accuracy:
            best_accuracy = acc
            best_params = (k, metric)

print("Best hyperparameters:", best_params, "->", best_accuracy)

Tracing this by hand against the neighbor lists worked out earlier, the output is:

k=1, metric=euclidean: accuracy=1.00
k=1, metric=manhattan: accuracy=1.00
k=3, metric=euclidean: accuracy=1.00
k=3, metric=manhattan: accuracy=1.00
k=5, metric=euclidean: accuracy=1.00
k=5, metric=manhattan: accuracy=1.00
k=7, metric=euclidean: accuracy=1.00
k=7, metric=manhattan: accuracy=1.00
k=9, metric=euclidean: accuracy=0.50
k=9, metric=manhattan: accuracy=0.50
Best hyperparameters: (1, 'euclidean') -> 1.0

The chart below shows the same result visually — flat, perfect accuracy through k = 7, then a sharp drop at k = 9 as the model starts defaulting to the majority class:

Common Misconception: "The Highest Validation Accuracy Is Always the Right Choice"

Look again at the grid search output: k = 1, k = 3, k = 5, and k = 7 all tie at a perfect 1.00. Our code picks (1, 'euclidean') simply because it was the first combination it tried that reached the top score — not because k = 1 is genuinely the most trustworthy setting. This is the misconception to watch for: a single validation score, especially from a small validation set of only 4 students, is not proof that a hyperparameter is truly the best one. We already established earlier in this chapter that small k values are the most fragile to noisy individual data points. Our validation set of 4 students simply did not happen to contain a case that exposed that fragility — a different set of 4 students might have. In real practice, when validation scores tie, engineers usually lean toward the more robust, moderately larger value rather than the smallest one, and they gain more confidence using k-fold cross-validation: split the labelled data into, say, 5 equal chunks, repeat the validation process 5 times (each time holding out a different chunk), and average the 5 accuracy scores per hyperparameter combination. This uses the data more thoroughly and reduces the chance that one lucky or unlucky split makes a fragile hyperparameter look artificially good. Be careful of a naming collision here: the "k" in k-Nearest-Neighbors (how many neighbors vote) and the "k" in k-fold cross-validation (how many chunks you split your data into) are two completely unrelated uses of the same letter.

Beyond Grid Search: When the Grid Gets Too Big

Grid search worked cleanly here because we only searched 5 values of k times 2 metrics — 10 combinations total, each cheap to check. Real models often need many more hyperparameters tuned together. Suppose a more advanced model has 4 hyperparameters, each with 10 candidate values worth trying. A full grid search would require 10 × 10 × 10 × 10 = 10,000 combinations, and each one means retraining and re-evaluating the model from scratch. If each run took even one minute, that is over 166 hours before you have accomplished anything else. Random search handles this by picking a fixed budget of random combinations — say, 200 — instead of exhaustively trying every cell in the grid. It sounds less thorough, and in a strict sense it is, but it usually finds nearly-as-good hyperparameters far faster, because in most real models only one or two hyperparameters actually matter a lot for performance, and randomly sampling explores a wider range of values for those important ones than a rigid, evenly-spaced grid does.

Practice: Test Your Understanding

1. Without re-reading the table, try to recall: what is the exact range of Attendance% values in the training table at the start of this chapter, and what is the range of StudyHours values? Why does this difference in scale matter for a distance-based hyperparameter like k in KNN?
2. In the grid search results, why did k = 9 predict "Pass" incorrectly for Neha, even though her nearest neighbors up to k = 7 were mostly Fail?
3. If you wanted to grid search over k ∈ {1, 3, 5} and metric ∈ {"euclidean", "manhattan"}, how many total hyperparameter combinations would grid search need to evaluate?
4. A classmate says: "I'll just always set k = 1, since the single closest match is obviously the most similar and therefore the most trustworthy neighbor." Using the concept of noise/outliers from this chapter, explain what is wrong with this reasoning.

Answers

1. Attendance ranges from 40 (Amit) to 90 (Hari) — a spread of exactly 50 percentage points. StudyHours ranges from 2 to 9 — a spread of only 7. Because Euclidean and Manhattan distance both add up raw numeric differences, a modest change in Attendance contributes far more to the total distance than an equally meaningful change in StudyHours, since StudyHours cannot even vary by more than 7 across the whole dataset. In practice, this means Attendance would dominate every distance calculation, quietly overpowering StudyHours — which is why real KNN pipelines usually rescale features onto a comparable range before tuning k.
2. Neha's 7 nearest neighbors split 4 Fail to 3 Pass, correctly predicting Fail. But there are only 4 Fail students in the entire 10-student training set — Neha's 8th and 9th nearest neighbors, pulled in only because k grew to 9, had to be Pass students (there was no Fail student left to include). That flipped the vote to 5 Pass vs. 4 Fail, overriding the correct local pattern with the dataset's overall majority class.
3. 3 values of k × 2 metrics = 6 total combinations.
4. The single nearest neighbor could easily be an outlier or a noisy, unusual case — a student who failed for a reason unrelated to the two measured features, or one whose result was recorded slightly inaccurately. Relying on exactly one vote gives that single point complete control over the prediction, with no other neighbor able to outvote it if it happens to be wrong. A slightly larger k lets nearby neighbors "check" each other through majority vote, which is more resistant to any single noisy data point — this is precisely why very small k is associated with overfitting.

Summary

• A parameter is learned automatically from data during training; a hyperparameter is a setting you choose before training or prediction begins, and no formula can compute its "correct" value for you.
• In KNN, k (how many neighbors vote) and the distance metric (Euclidean or Manhattan) are both hyperparameters — the same running example generalizes to max_depth in decision trees, the number of clusters in k-Means, and the learning rate in gradient-based models.
• Too small a k makes predictions fragile to noisy individual data points (overfitting); too large a k makes predictions collapse toward the dataset's overall majority class regardless of local pattern (underfitting) — we proved this directly with Neha's and Suresh's flipped predictions at k = 9.
• Hyperparameters must be tuned on a separate validation set, never on the training data itself — otherwise something like k = 1 will trivially score 100% by matching every training point to itself at distance zero.
• Grid search exhaustively tests every combination of candidate hyperparameter values on the validation set and keeps the best-scoring combination; random search samples a fixed budget of combinations instead, trading completeness for speed when the number of combinations grows too large to test exhaustively.
• A tied or top validation score is not automatic proof of the best hyperparameter, especially on a small validation set — k-fold cross-validation, which averages accuracy across several different training/validation splits, gives a more reliable picture.