Ensemble Methods: Stacking and Blending

Two tutors are trying to predict how you will score in your upcoming Class 9 Half-Yearly Mathematics exam, out of 100. Tutor A only looks at your attendance record and how many practice problems you finished this term. Tutor B only looks at the average of your last five unit-test scores. You ask them both, and you get two different numbers. Which one do you trust? Do you split the difference and average them? Or is there a smarter way to combine two imperfect opinions into one better prediction?

This exact question — how do you combine several different models' predictions into a single, better prediction — is what stacking and blending are built to answer. You have already seen two other ways of combining models: bagging (train many similar models on random subsets of data and vote or average, like a Random Forest) and boosting (train models one after another, each one fixing the previous one's mistakes, like AdaBoost or Gradient Boosting). Stacking and blending are a third, different idea: instead of voting or averaging by a fixed rule, you train another model whose entire job is to learn how much to trust each of your other models.

Why Averaging Isn't Always Smart: A Worked Example

Let's make the tutor scenario numeric. Suppose we collect data from 5 students whose actual exam scores we already know, along with what Tutor A (attendance-based) and Tutor B (past-average-based) predicted for each of them before the results came out.

• Actual scores: 82, 65, 90, 55, 74
• Tutor A's predictions: 90, 55, 78, 68, 62
• Tutor B's predictions: 84, 63, 88, 58, 76

To measure how good each tutor is, we use Mean Absolute Error (MAE) — the average of how far off each prediction was, ignoring whether it was too high or too low.

actual  = [82, 65, 90, 55, 74]
tutor_A = [90, 55, 78, 68, 62]
tutor_B = [84, 63, 88, 58, 76]

def mean_absolute_error(y_true, y_pred):
    errors = [abs(a - p) for a, p in zip(y_true, y_pred)]
    return sum(errors) / len(errors)

print(mean_absolute_error(actual, tutor_A))   # 11.0
print(mean_absolute_error(actual, tutor_B))   # 2.2

Let's trace this by hand for Tutor A: |82-90|=8, |65-55|=10, |90-78|=12, |55-68|=13, |74-62|=12. Sum = 55, divided by 5 students = MAE of 11.0. For Tutor B: |82-84|=2, |65-63|=2, |90-88|=2, |55-58|=3, |74-76|=2. Sum = 11, divided by 5 = MAE of 2.2. Tutor B is clearly the much better predictor here.

Now here is the misconception to watch for: many students assume that combining two models by simple averaging always produces a result at least as good as the better of the two. Let's test that belief directly. If we average Tutor A and Tutor B's predictions with equal weight (0.5 each):

simple_avg = [0.5*a + 0.5*b for a, b in zip(tutor_A, tutor_B)]
print(simple_avg)
# [87.0, 59.0, 83.0, 63.0, 69.0]
print(mean_absolute_error(actual, simple_avg))  # 6.2

Tracing this: student 1 gives 0.5(90)+0.5(84) = 45+42 = 87, against an actual of 82, an error of 5. Doing this for all five students and averaging the errors (5, 6, 7, 8, 5) gives an MAE of 6.2. Compare the three numbers: Tutor A alone scores 11.0, Tutor B alone scores 2.2, and the "obviously safe" 50-50 average scores 6.2 — worse than Tutor B on its own. Because Tutor A is so much noisier, blindly giving it equal say drags the combined prediction away from the more accurate Tutor B. Averaging is not automatically an improvement; it only helps when the models being combined are reasonably close in skill, or when their errors cancel out. When one model is clearly stronger, a fixed 50-50 rule can actively hurt you.

What if, instead of guessing 50-50, we let the combination trust Tutor B more, say 20% Tutor A and 80% Tutor B?

weighted = [round(0.2*a + 0.8*b, 2) for a, b in zip(tutor_A, tutor_B)]
print(weighted)
# [85.2, 61.4, 86.0, 60.0, 73.2]
print(round(mean_absolute_error(actual, weighted), 2))  # 3.32

Tracing student 1: 0.2(90)+0.8(84) = 18+67.2 = 85.2, error 3.2. Across all five students the errors are 3.2, 3.6, 4.0, 5.0, 0.8, giving an MAE of 3.32 — much closer to Tutor B's solo performance, and clearly better than the naive 50-50 average's 6.2. The weights 0.2 and 0.8 weren't magic; they were a reasonable guess based on knowing Tutor B was roughly five times more accurate than Tutor A. This raises the natural next question: instead of us guessing the weights by trial and error, can a model learn the best weights itself, directly from data? That is precisely what stacking automates.

Formalizing the Idea: Base Learners and a Meta-Learner

In stacking, we organize models into two layers:

• Level-0 (base learners): Several different models trained on the original features to solve the same prediction problem — for example, a Decision Tree, a k-Nearest Neighbours model, and a Linear Regression model, all trying to predict the same exam score from the same student data.
• Level-1 (the meta-learner): A new, usually simple, model whose input features are the predictions made by the Level-0 models, and whose target is still the real answer. The meta-learner's job is to learn a function like "when Tutor A says 90 and Tutor B says 84, and historically Tutor B tends to be right more often in cases like this, output something close to 85" — a learned combination rule, not a fixed one.

The tutor-weighting example above, where we tried 0.5/0.5 and then 0.2/0.8, is a simplified version of exactly what a meta-learner does: it is a small regression problem where the base models' predictions are the inputs and the true score is the output. A real stacking system would run something like a linear regression on the pairs (Tutor A's prediction, Tutor B's prediction) against the actual scores, and the regression would solve for the best weights mathematically, rather than us trying 0.2/0.8 by hand. In practice the meta-learner doesn't have to be linear at all — it can itself be a small decision tree or even another ensemble — but a simple linear or logistic regression is the most common choice because with only a handful of base-model predictions as input, a complex meta-learner tends to overfit.

The Trap: Why You Can't Train the Meta-Learner on the Same Data the Base Models Trained On

Here is the second, more serious misconception: it seems natural to train your base models on the training set, get their predictions on that same training set, and feed those predictions straight into the meta-learner. This is wrong, and it is the single most common bug in a first attempt at stacking.

The problem is that a model's predictions on data it was trained on are usually far too optimistic — a Decision Tree, if left deep enough, can memorize its training data and predict it almost perfectly, but that says nothing about how well it will predict new students it hasn't seen. If the meta-learner is shown these overly optimistic, "already memorized" predictions, it will learn to trust the base models much more than it should, and the whole stacked system will perform beautifully on data it has already seen and badly on new data. This is a form of data leakage: information about the true answer leaks into the meta-learner's training signal through an overfit base model.

The fix is to make sure every prediction fed to the meta-learner is a prediction the base model made on data it did not use for training. This is done using k-fold cross-validation, producing what are called out-of-fold predictions.

How Real Stacking Works, Step by Step

Suppose we have 4 students with "hours studied per week" (x) and actual score (y): x = [2, 4, 6, 8], y = [40, 58, 68, 85]. We'll generate honest, out-of-fold predictions from a single base model (a simple straight-line fit) using 2-fold cross-validation — splitting the 4 students into Fold A (x=2,4) and Fold B (x=6,8).

Round 1: Train the base model only on Fold B (x=6, y=68 and x=8, y=85). Fitting a line through these two points: slope = (85-68)/(8-6) = 8.5, and intercept solves 68 = 8.5(6) + b, giving b = 17. So the model is prediction = 8.5x + 17. Now use this model, which has never seen Fold A, to predict Fold A: at x=2, prediction = 8.5(2)+17 = 34; at x=4, prediction = 8.5(4)+17 = 51.

Round 2: Train the base model only on Fold A (x=2, y=40 and x=4, y=58). Slope = (58-40)/(4-2) = 9, intercept solves 40 = 9(2)+b, giving b = 22. Model: prediction = 9x + 22. Predict Fold B, which this version has never seen: at x=6, prediction = 9(6)+22 = 76; at x=8, prediction = 9(8)+22 = 94.

Collecting the results, every student now has one honest, out-of-fold prediction: x=2 → 34, x=4 → 51, x=6 → 76, x=8 → 94, compared to actual y = 40, 58, 68, 85. Notice these predictions are believable, imperfect guesses — not suspiciously perfect memorized answers — because in each case the model predicting a student had never trained on that student. These four (out-of-fold prediction, actual score) pairs are exactly the honest training data the meta-learner is allowed to use. With, say, three base models instead of one, each student would get three such out-of-fold predictions, forming a three-column table that becomes the meta-learner's input features.

Once the meta-learner is trained this way, there's one more step: the base models are then retrained on the full training set (all the data, no folds held out) so that when a genuinely new student arrives, each base model makes one final prediction, those predictions are fed into the trained meta-learner, and out comes the final answer.

Blending: The Faster, Simpler Cousin

Stacking's k-fold procedure gives every training example an honest out-of-fold prediction, but it costs a lot of computation — each base model must be trained k separate times just to build the meta-learner's training set, and then once more on the full data. Blending is a lighter-weight version of the same idea that trades some data-efficiency for simplicity and speed.

In blending, you split your training data into two plain chunks up front — say 70% and 30%. The base models are trained only once, on the 70% chunk. They then predict, only once, on the 30% holdout chunk. Those predictions on the holdout chunk, together with the real answers for the holdout students, become the meta-learner's training data. There is no rotating of folds and no retraining loop.

The trade-off is real: because blending's base models only ever see 70% of the data, and the meta-learner only ever learns from one particular 30% slice, results can be more sensitive to exactly how that one split happened to fall — a student who is unusually hard to predict landing in the holdout chunk by chance can noticeably skew the meta-learner's training. Stacking's k-fold approach uses every example for both training and out-of-fold prediction at some point, which tends to be more stable, at the cost of extra computation. Kaggle competition winners, who often have days of compute time and want to squeeze out every fraction of accuracy, lean toward k-fold stacking, sometimes with several stacked layers. Teams that need a quick, "good enough" ensemble on a laptop often reach for blending instead.

Stacking and Blending vs. What You Already Know

It helps to place stacking and blending next to bagging and boosting, since a common exam-style confusion is treating all ensemble methods as "basically the same thing."

• Bagging (e.g., Random Forest): many copies of the same kind of model, trained on random subsets of the data, combined by a fixed rule — majority vote for classification, simple average for regression. No learning happens in the combination step.
• Boosting (e.g., AdaBoost, Gradient Boosting): models are trained one after another, in sequence, and each new model focuses on the mistakes of the ones before it. The final combination is typically a weighted sum, but the weights come from the boosting algorithm's schedule, not from a separately trained model.
• Stacking / Blending: base models can be completely different kinds of models (a tree, a k-NN, a regression, even a neural network, all at once), trained independently and usually in parallel, and the combination itself is done by a trained model — the meta-learner — which studies the base models' track record and learns how much to trust each one, possibly differently in different situations.

That last point is worth dwelling on: a meta-learner doesn't have to use fixed weights everywhere. If Tutor A (attendance-based) tends to be more reliable specifically for students with high attendance, while Tutor B is more reliable for students with erratic attendance, a sufficiently expressive meta-learner can learn that pattern and trust each tutor more in the situations where they are historically stronger — something a single fixed averaging rule can never do.

Why Diversity Among Base Models Matters

Stacking only pays off when the base models make different kinds of mistakes. If Tutor A and Tutor B were both attendance-based, they would tend to be wrong about the same students in the same direction, and no meta-learner could extract anything extra from combining them — averaging two nearly identical opinions doesn't create a third, better opinion. The value in our worked example came precisely from Tutor A and Tutor B using genuinely different information (attendance versus past scores), so their errors were at least partly independent. This is why real stacking systems deliberately mix model families — a tree-based model, a distance-based model like k-NN, and a linear model — rather than three slightly different decision trees, which is closer to what bagging already does.

Common Misconceptions, Corrected

• "Combining models by averaging always helps." False, as our worked example showed — a 50-50 average scored worse (MAE 6.2) than the better model alone (MAE 2.2). Combination only helps when done with informed weights or when the models are reasonably comparable in skill.
• "Stacking and blending are the same technique with two names." They share the two-layer idea, but stacking generates meta-features through rotating k-fold out-of-fold predictions, while blending uses one fixed holdout split. This changes both the computational cost and the stability of the result.
• "You can train the meta-learner on the base models' predictions on their own training data." This causes data leakage: base models look artificially accurate on data they've memorized, so the meta-learner learns to over-trust them, and the ensemble performs worse than expected on genuinely new data.
• "More base models is always better." Adding a fourth, fifth, or sixth base model that makes similar mistakes to existing ones adds computational cost without adding new information for the meta-learner to exploit. Diversity of model type matters more than sheer count.

Where This Shows Up Beyond the Classroom

Large-scale recommendation and ranking systems — the kind that decide what to show you next on a shopping app or a video platform — frequently combine several very different signal models (one based on your past behaviour, one based on what similar users liked, one based on item popularity trends) using exactly this stacked structure, because no single signal is reliable enough on its own and the right blend of signals shifts from user to user. In data science competitions, stacking with several layers of meta-learners is one of the most common techniques used by top-performing solutions, precisely because it lets a competitor combine very different modelling approaches into one system that is more accurate than any single approach alone.

Check Your Understanding

1. Two base models predict a student's Science score as 70 and 90. The true score is 85. Compute the absolute error of a simple 50-50 average, and then of a weighted combination using 0.25 and 0.75 (favouring the second model). Which is closer, and by how much?
2. Explain, in your own words, why training a meta-learner directly on base models' predictions on their own training set is a mistake, and name the concept this mistake is an example of.
3. You are given 6 training examples and asked to build out-of-fold predictions using 3-fold cross-validation. How many times will each base model need to be trained during this process (not counting the final retrain on all the data)?
4. A classmate says, "Blending is strictly worse than stacking, so nobody should ever use it." Give one legitimate reason someone might choose blending over stacking anyway.
5. Why does stacking generally benefit more from combining a Decision Tree, a k-NN model, and a Linear Regression model together than from combining three Decision Trees of slightly different depths?

Summary

Stacking and blending combine predictions from several different base models by training a separate meta-learner to learn the best way to combine them, rather than relying on a fixed rule like majority vote (bagging) or a preset weighting schedule (boosting). The key numerical lesson is that naive averaging can perform worse than your best single model when the base models differ substantially in skill — a trained meta-learner avoids this by learning appropriate weights, or even situation-dependent weights, directly from data. Getting this right requires generating honest, out-of-fold predictions for the meta-learner to train on: stacking does this rigorously through rotating k-fold cross-validation, while blending uses a single, faster train-holdout split at some cost to stability. The technique works best when base models are genuinely diverse in how they make mistakes, since a meta-learner can only extract value from models that fail in different ways.