Random Forests: Ensemble Learning Power

Why the BCCI Doesn't Pick the Playing XI With One Selector

India's cricket selection committee is never a single person. It is a panel — usually five selectors — who each independently form a view on who should play, then vote. Why not just appoint the single sharpest cricket mind in the country and let that person decide alone? Because even a brilliant selector has blind spots: a bias toward players from their own state, a soft spot for a particular batting style, a bad day where one net session colors their judgment too heavily. A panel doesn't eliminate every individual's blind spots, but it makes it very unlikely that all five selectors share the exact same blind spot on the same day. Their mistakes tend to cancel out; their shared knowledge reinforces itself. The committee's combined decision is, on average, more reliable than any one selector's decision — even if that one selector is, individually, the best of the five.

This is the entire idea behind a Random Forest, applied to machine learning. Instead of training one decision tree and trusting it completely, you train a whole panel of decision trees — often a hundred or more — each one seeing the training data slightly differently, and then you let them vote. This chapter builds that idea from the ground up: what exactly makes each tree in the forest "see the data differently," why that matters mathematically and not just intuitively, and how the final vote is taken. We will do this with one small, fully worked dataset that you can recompute by hand, and with real Python code whose output is verified, not guessed.

The Dataset: Will a Student Pass?

Suppose a school (unofficially, and just for practice — no real school actually does this) tries to predict whether a Class 9 student will pass their unit test based on three yes/no signals: whether they studied more than 4 hours that week (StudyHours: High/Low), whether their attendance was 75% or above (Attendance: High/Low), and whether they slept well the night before the test (Sleep: Good/Poor). Here are eight students' records:

Student  StudyHours  Attendance  Sleep   Result
S1       High        High        Good    Pass
S2       High        High        Poor    Pass
S3       High        Low         Good    Pass
S4       High        Low         Poor    Fail
S5       Low         High        Good    Pass
S6       Low         High        Poor    Fail
S7       Low         Low         Good    Fail
S8       Low         Low         Poor    Fail

Four students passed, four failed — a perfectly balanced dataset. This is deliberate: it will let us do clean arithmetic while we build the ideas.

Quick Recap: How One Decision Tree Chooses Its Splits

A decision tree decides, at every node, which feature best separates the two outcomes, and it needs a number to compare candidate features objectively. The measure scikit-learn uses by default is called Gini impurity. For a group of examples with two classes, if p is the fraction that are "Pass," the Gini impurity is:

Gini = 1 - (p^2 + (1-p)^2)

A pure group (all Pass or all Fail) has Gini = 0 — nothing left to explain. A perfectly mixed 50/50 group has the maximum Gini for two classes. For all 8 students, 4 are Pass (p = 4/8 = 0.5), so:

Gini(all 8) = 1 - (0.5^2 + 0.5^2) = 1 - 0.5 = 0.5

To choose the root split, the tree tries each feature, splits the data into two groups, and computes the weighted Gini of the two resulting groups. Whichever feature drops the Gini the most (the biggest "impurity reduction") wins the root. Let's actually do this arithmetic for all three features, because the result is the whole point of this section.

Split on StudyHours. High = {S1, S2, S3, S4}: 3 Pass, 1 Fail, so Gini = 1 − (0.75² + 0.25²) = 1 − (0.5625 + 0.0625) = 0.375. Low = {S5, S6, S7, S8}: 1 Pass, 3 Fail — by symmetry, Gini = 0.375 too. Both groups have 4 students, so the weighted average is 0.375. Impurity reduction = 0.5 − 0.375 = 0.125.

Split on Attendance. High = {S1, S2, S5, S6}: 3 Pass, 1 Fail → Gini = 0.375. Low = {S3, S4, S7, S8}: 1 Pass, 3 Fail → Gini = 0.375. Weighted average = 0.375. Reduction = 0.125.

Split on Sleep. Good = {S1, S3, S5, S7}: 3 Pass, 1 Fail → Gini = 0.375. Poor = {S2, S4, S6, S8}: 1 Pass, 3 Fail → Gini = 0.375. Weighted average = 0.375. Reduction = 0.125.

All three features reduce impurity by exactly 0.125. This three-way tie is built into the dataset on purpose, and it is about to matter a great deal. A standard decision-tree algorithm cannot leave a tie unresolved — it must pick one feature as the root and commit to it, using some internal tie-breaking rule (often just "the first feature it checked"). I actually built this exact tree in scikit-learn's DecisionTreeClassifier to see what it does. It picked Attendance as the root:

Attendance <= High?
├─ No (Low):
│    Sleep <= Poor?
│    ├─ Yes → Fail   (covers S4, S8)
│    └─ No (Good):
│         StudyHours <= Low?  → Fail (S7)   High → Pass (S3)
└─ Yes (High):
     Sleep <= Poor?
     ├─ Yes: StudyHours <= Low? → Fail (S6)   High → Pass (S2)
     └─ No (Good): → Pass   (covers S1, S5)

This tree gets every single training row correct — 100% training accuracy, verified by running it. That sounds impressive, but it should worry you a little. With 3 yes/no features there are only 2×2×2 = 8 possible students, and we happen to have all 8 in our table with no repeats and no noise. A tree that grows until every leaf is pure will always be able to memorise a table like this perfectly. The real test is not "does it fit the students we already have?" — it's "will it correctly judge a ninth student it has never seen?" A tree deep enough to fit every quirk of its training data, including quirks that are just coincidence, is overfit, and that is the specific weakness Random Forests are built to fix.

The Instability Problem: A Tie Means the Tree Is Fragile

Go back to that three-way tie. If StudyHours, Attendance, and Sleep are all equally good root candidates on this data, then the tree we got — rooted at Attendance — is not "the correct tree." It's just the one scikit-learn's tie-breaking rule happened to produce. If a slightly different sample of students had nudged the Gini numbers by even a tiny amount (say, one student's sleep quality had been recorded differently), a completely different feature could have won the root, and the entire tree below it would branch differently, even though it might still fit the training data just as perfectly. A single decision tree's structure can be highly sensitive to small changes in the data it's trained on. Statisticians call this high variance: retrain on a slightly different sample, get a meaningfully different model. This instability, more than any single wrong prediction, is the core problem a Random Forest is designed to solve.

Building the Forest: Two Separate Sources of Randomness

A Random Forest, introduced by the statistician Leo Breiman in 2001, is an ensemble — a collection of decision trees whose individual predictions are combined into one final answer. But it doesn't just train the same tree seven times and hope for different results; identical training data with no randomness would produce identical trees every time. Instead, each tree in the forest is deliberately made to see the world slightly differently, using two distinct mechanisms:

1. Bagging (Bootstrap AGGregatING). Instead of training every tree on all 8 students, each tree gets its own bootstrap sample: 8 students drawn at random with replacement from the original 8. "With replacement" means the same student can be picked more than once, and — as a direct consequence — some students may not be picked at all. Each tree therefore trains on a slightly different, resampled version of the class.
2. Random feature subsets at each split. When a tree is deciding how to split a node, instead of considering all 3 features every time, it is only allowed to consider a random subset of them (in our forest, 2 out of 3, chosen freshly at every split). This forces trees to sometimes build splits around a feature they would never have chosen if the strongest feature were available.

Let's see both mechanisms in action with real, verified output. I trained a forest of 7 trees (a small number, chosen deliberately so we can look at every single one) with max_features=2 and a depth limit of 3. Here is exactly which students landed in each tree's bootstrap sample, which students were left out (called out-of-bag, or OOB), and which feature each tree picked as its root:

Tree 1: sample = S8,S2,S1,S6,S2,S3,S1,S4  | OOB = S5,S7   | root: StudyHours
Tree 2: sample = S6,S4,S2,S4,S1,S7,S4,S3  | OOB = S5,S8   | root: Attendance
Tree 3: sample = S5,S2,S3,S5,S2,S7,S2,S4  | OOB = S1,S6,S8| root: Attendance
Tree 4: sample = S8,S2,S6,S8,S3,S4,S2,S5  | OOB = S1,S7   | root: Sleep
Tree 5: sample = S4,S2,S5,S6,S4,S7,S6,S1  | OOB = S3,S8   | root: Attendance
Tree 6: sample = S2,S1,S8,S8,S3,S3,S6,S7  | OOB = S4,S5   | root: StudyHours
Tree 7: sample = S8,S1,S6,S3,S3,S8,S8,S4  | OOB = S2,S5,S7| root: StudyHours

Look at what happened to that three-way tie. Once each tree sees its own resampled version of the class (some students duplicated, some missing), the exact tie is broken differently every time: three trees pick StudyHours as the root, three pick Attendance, one picks Sleep. No two of these trees are identical, and several are structurally quite different from the single tree we built on the full dataset. This is bagging visibly doing its job — creating genuine diversity out of one dataset.

Correcting a Common Misconception

Here is a mistake worth naming directly, because it appears in a lot of simplified explanations: "A Random Forest is just Bagging applied to Decision Trees." This is incomplete, and the missing piece is exactly mechanism #2 above. Bagging (resampling rows) is a general technique that works with any model, not just trees — you could bag linear regressions or k-nearest-neighbours just as easily. If you only bagged decision trees, without the random feature-subset restriction, you would get an ensemble method called Bagged Trees — a real, older, and less powerful cousin. On data with one dominant, obviously-best feature, bagging alone tends to make nearly every tree pick that same feature as its root anyway, because it is still usually the best choice even after resampling. The trees end up correlated, which limits how much their errors cancel out when you average them. A Random Forest's specific innovation is to also randomly withhold features at each split, forcing trees to sometimes build around a weaker signal and discover different, still-valid patterns. This is the extra decorrelation that gives Random Forest its name and its edge over plain Bagged Trees.

Taking the Vote: How the Forest Reaches a Verdict

For a classification forest with B trees, each tree T_i makes its own prediction on a new input x, and the forest's final answer is whichever class gets the most votes:

Forest(x) = mode{ T1(x), T2(x), T3(x), ..., TB(x) }

("Mode" just means "the most frequent value" — plain majority vote.) Let's watch this happen concretely, using our 7 real trees on student S4: StudyHours = High, Attendance = Low, Sleep = Poor, whose true recorded result is Fail. Tracing S4 through each tree's actual split rules gives:

Tree 1 → Fail    Tree 2 → Fail    Tree 3 → Fail    Tree 4 → Fail
Tree 5 → Fail    Tree 6 → Pass    Tree 7 → Fail

Vote tally:  Fail = 6,  Pass = 1
Forest's final answer: FAIL  ✓ (matches the true result)

Notice Tree 6 got it wrong. Look back at its structure: because of the particular bootstrap sample it trained on (which happened to make StudyHours alone perfectly separate Pass from Fail within that resample), Tree 6 grew into a lazy one-split tree — "High StudyHours → Pass, otherwise → Fail" — and it never learned that low attendance and poor sleep can drag a high-study student into failing. On its own, Tree 6 would confidently give a wrong answer for S4. But it is only one voice among seven. The other six trees, having trained on different resamples, caught the pattern that low attendance combined with poor sleep matters, and their combined vote overruled Tree 6's mistake. This is not a lucky coincidence in this one example — it is the mathematical reason ensembles work: individual trees make individual mistakes, but as long as those mistakes aren't all pointing the same wrong direction, averaging (or voting) cancels a good portion of them out. This is the direct payoff of the row-resampling and feature-randomness we built up earlier — it's what makes each tree's mistakes different enough to cancel.

Here's the SVG Diagram of Exactly This Vote

Why Feature Randomness Matters More Than It Looks

It's tempting to think bagging alone should already be enough — after all, the 7 bootstrap trees above already had 6 different roots between them just from row resampling before we even discuss feature restriction. But that diversity was helped along by our engineered three-way Gini tie. In most real datasets there is no tie: one feature is clearly, robustly the best predictor, and it stays the best predictor across almost every resampled version of the data too. If every tree in the forest is still allowed to freely choose from all features at every split, most trees will independently rediscover that same dominant feature as their root, and their predictions will end up highly correlated — meaning when one tree is wrong, many others tend to be wrong in the same way, for the same reason. Restricting each split to a random subset of features (2 out of 3 here; for larger datasets a common rule of thumb, following Breiman's original recommendation, is to use roughly the square root of the total number of features for classification tasks) deliberately blinds some trees to the obvious answer at some nodes, forcing them to build around a secondary signal instead. The forest ends up with trees that disagree in more varied and independent ways, so their vote genuinely averages out mistakes rather than just repeating the same one loudly seven times.

Feature Importance: Which Clue Did the Forest Lean On?

Because a Random Forest builds many trees and records, at every split in every tree, how much that split reduced Gini impurity, it can report an overall feature importance score: add up each feature's total impurity reduction across every split in every tree, then normalise so all features' scores sum to 1. Running this on our 7-tree forest gave:

StudyHours: 0.533
Attendance: 0.241
Sleep:      0.226

Remember that on the full 8-student dataset, all three features tied exactly on impurity reduction at the very first split. Yet once you build many trees on many resamples with restricted feature choices, StudyHours ends up mattering noticeably more overall — because across the different bootstrap samples and the different orders in which features became "available" at each node, StudyHours happened to be selected more often and at more decisive (higher-impurity) points in the trees. This is a genuinely useful, data-driven output: it tells you which signals the forest actually leaned on, not just which one theoretically looked strongest on the original untouched table.

Out-of-Bag Rows: A Validation Set You Get for Free

Look again at the bootstrap table above. Every tree left a few students out — Tree 1 never saw S5 or S7, Tree 3 never saw S1, S6, or S8, and so on. Those left-out rows are called out-of-bag (OOB) samples for that tree, and they're valuable: since that particular tree never trained on them, you can use them to test it, almost like a free held-out validation set, without setting aside any data in advance. Across the whole forest, each student gets evaluated only by the subset of trees that never saw them during training, and comparing those OOB predictions to the true labels gives an estimate of how the forest would perform on genuinely new students — called the OOB score. There's a neat, well-known fact behind why roughly a third of the rows go missing from any one bootstrap sample: when you draw n items with replacement from a set of n, the chance any specific item is never picked approaches 1/e ≈ 0.368 as n grows — so on average about 36.8% of rows are left out of each tree's sample, and about 63.2% are included (with repeats). With our tiny 8-row dataset the fractions bounce around more (our trees left out between 2 and 3 of 8 students, i.e. 25%–37.5%), which is close to that expected rate but noisier because 8 is a very small sample size. When I actually computed the OOB score for a larger 50-tree version of this exact forest, it came out to 0.375 — correctly reflecting that with only 8 total training rows, OOB estimates are too data-starved to be trustworthy. This is an honest limitation, not a flaw in the method: OOB scoring becomes genuinely reliable only once your dataset has enough rows that "roughly a third, left out per tree" still means a reasonably sized sample.

Correcting a Second Misconception: "More Trees Always Means More Accuracy"

Adding more trees to a Random Forest reduces the variance of its predictions — averaging over more independent voters makes the final vote more stable and less sensitive to which particular bootstrap samples happened to get drawn. But it does not indefinitely improve accuracy. Beyond some point (often a few hundred trees, depending on the dataset), adding more trees barely changes the vote at all — you're just averaging in more copies of a similar signal, with diminishing returns and rising computation cost for almost no benefit. Crucially, more trees also cannot fix problems that come from the trees themselves being too weak or the data being unrepresentative — if every tree is restricted to depth 1 and the true pattern needs depth 3 to express, no number of shallow trees voting together will discover it, because they're all failing to capture the same essential structure. More trees cure instability (high variance); they do not cure an underpowered underlying model (high bias) or bad data.

Classification vs Regression Forests

Everything above used RandomForestClassifier, which is for categorical outputs like Pass/Fail, and combines trees by majority vote. For predicting a continuous number instead — say, a student's actual marks out of 100, or expected rainfall in millimetres — you would use RandomForestRegressor. The trees and randomisation mechanisms (bagging plus random feature subsets) work identically; the only change is how the forest combines the trees' outputs: instead of a majority vote, it takes the plain average of all the trees' predicted numbers.

The Code, Exactly As Run

Here is the actual code used to produce every verified number in this chapter — the dataset, the forest, and the S4 vote trace:

import numpy as np
from sklearn.ensemble import RandomForestClassifier

# Columns: StudyHours, Attendance, Sleep (1 = High/Good, 0 = Low/Poor)
X = np.array([
    [1,1,1],  # S1
    [1,1,0],  # S2
    [1,0,1],  # S3
    [1,0,0],  # S4
    [0,1,1],  # S5
    [0,1,0],  # S6
    [0,0,1],  # S7
    [0,0,0],  # S8
])
y = np.array([1,1,1,0,1,0,0,0])  # Pass=1, Fail=0

clf = RandomForestClassifier(
    n_estimators=7, max_features=2, max_depth=3, random_state=7
)
clf.fit(X, y)

S4 = np.array([[1,0,0]])
votes = [tree.predict(S4)[0] for tree in clf.estimators_]
print(votes)                       # [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0]
print(clf.predict(S4))             # [0]  -> Fail, matches the true label
print(np.round(clf.feature_importances_, 3))
                                    # [0.533 0.241 0.226]

Every number in the comments above is the real, executed output — this is not a hypothetical trace. The forest correctly predicts Fail for S4 by a 6–1 vote, and the feature importances match what was discussed earlier.

Active Recall: Check Your Understanding

1. Using the original 8-student table, suppose a new bootstrap sample for "Tree 8" is: S1, S1, S3, S5, S6, S6, S7, S8. Which students are out-of-bag for Tree 8? (List them.)
2. For that same Tree 8 sample, count Pass vs Fail: S1(Pass) appears twice, S3(Pass), S5(Pass), S6(Fail) appears twice, S7(Fail), S8(Fail). Compute the Gini impurity of this bootstrap sample as a whole, using Gini = 1 − (p² + (1−p)²).
3. A classmate says: "Random Forest is just training the same decision tree seven times." Explain, in one or two sentences, exactly what is wrong with that statement and name the two mechanisms that make the trees different from each other.
4. In our worked S4 example, Tree 6 voted incorrectly. Explain, using the bootstrap sample it trained on, why Tree 6 grew into a tree that only checked StudyHours.
5. True or False, with a one-line justification: "Training a Random Forest with 5,000 trees instead of 500 will always give noticeably higher accuracy."

Answers: (1) S2 and S4 are out-of-bag (every other student appears at least once in the sample). (2) 4 Pass out of 8, p = 0.5, so Gini = 1 − (0.25 + 0.25) = 0.5 — this particular resample happens to still be perfectly balanced. (3) It's wrong because without added randomness, identical training data plus an identical algorithm produces identical trees every time; the two mechanisms are bagging (each tree trains on its own bootstrap-resampled rows) and random feature subsets (each split only considers a random subset of features, not all of them). (4) Tree 6's bootstrap sample (S2,S1,S8,S8,S3,S3,S6,S7) happened to be a set of rows where StudyHours alone perfectly separated Pass from Fail, so the algorithm found zero remaining impurity reduction available from Attendance or Sleep and stopped after one split — it wasn't "wrong" on its own training data, it just trained on a resample where the shortcut worked and never encountered a case (like the true S4) that broke it. (5) False — beyond a certain point (often a few hundred trees), additional trees mainly reduce small amounts of remaining variance and cost more compute for negligible accuracy gain; they cannot fix an underlying weak or biased model.

Summary

• A single decision tree can perfectly memorise its training data (0% training error) while still being unreliable on new data — this is overfitting, driven by the tree's high sensitivity (variance) to the exact rows it was trained on.
• A Random Forest is an ensemble of many decision trees whose predictions are combined by majority vote (classification) or averaging (regression).
• Each tree is decorrelated from the others through two distinct mechanisms: bagging (each tree trains on its own bootstrap sample — rows drawn with replacement) and random feature subsets (each split only considers a randomly chosen subset of features).
• Random Forest ≠ Bagging of Decision Trees; the feature-subsampling step is what specifically makes it a Random Forest and is what further decorrelates trees beyond what row-resampling alone achieves.
• Rows left out of a tree's bootstrap sample are called out-of-bag (OOB) samples and give a free estimate of generalisation performance — reliable on large datasets, noisy on tiny ones.
• Feature importance is computed from the total impurity reduction each feature contributes across all splits in all trees.
• More trees reduce variance and stabilise predictions but have diminishing returns; they cannot fix a model whose individual trees are too weak or whose data is unrepresentative.

Think About It

Think about this: How would you explain random forests: ensemble learning power to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.