Decision Trees and Random Forests: Interpretable Machine Learning

Twenty Questions, Played by a Machine

Play the game "Twenty Questions" with anyone who is good at it, and you will notice something: their first question is never narrow. A skilled player opens with something like "Is it alive?" or "Is it bigger than a microwave?" — a question that roughly splits the entire universe of possible answers in half. A weak player wastes an early question on "Is it a stapler?", which almost certainly gets a "no" and eliminates exactly one possibility out of thousands. The skilled player is, without knowing the name for it, solving an optimization problem: out of all the questions available, which one shrinks my uncertainty the most?

A decision tree is this exact strategy, automated. Given a pile of past examples with known outcomes, it searches through every possible question it could ask about the data, measures precisely how much each one would reduce uncertainty, asks the best one, and then repeats the process separately inside each resulting group — recursively — until the groups are "settled" enough to just announce an answer. A random forest, which we build toward in the second half of this chapter, is what happens when you stop trusting the outcome of one game of Twenty Questions and instead play it hundreds of times with slightly different decks of cards, then let the results vote. Both ideas sound simple. Making "shrinks my uncertainty the most" and "settled enough" mathematically precise is where the real content of this chapter lives.

The Dataset We Will Use Throughout

To keep every number in this chapter checkable by hand, we will use one small, fixed dataset for everything — entropy, information gain, Gini impurity, bagging, and random forests. It records 14 days at a school, each described by the day's weather, and whether outdoor cricket practice actually went ahead.

Day  Outlook   Temperature  Humidity  Wind    Practice?
D1   Sunny     Hot          High      Weak    No
D2   Sunny     Hot          High      Strong  No
D3   Overcast  Hot          High      Weak    Yes
D4   Rain      Mild         High      Weak    Yes
D5   Rain      Cool         Normal    Weak    Yes
D6   Rain      Cool         Normal    Strong  No
D7   Overcast  Cool         Normal    Strong  Yes
D8   Sunny     Mild         High      Weak    No
D9   Sunny     Cool         Normal    Weak    Yes
D10  Rain      Mild         Normal    Weak    Yes
D11  Sunny     Mild         Normal    Strong  Yes
D12  Overcast  Mild         High      Strong  Yes
D13  Overcast  Hot          Normal    Weak    Yes
D14  Rain      Mild         High      Strong  No

Out of 14 days, practice happened on 9 and was cancelled on 5. Four features are available to predict the outcome: Outlook (Sunny, Overcast, Rain), Temperature (Hot, Mild, Cool), Humidity (High, Normal), and Wind (Weak, Strong). This particular dataset is a classic in machine-learning teaching — you will see it again if you study this topic further — precisely because it is small enough to compute by hand yet rich enough to show every idea in this chapter honestly, with no shortcuts taken.

Anatomy of a Decision Tree

Before building one, fix the vocabulary. A decision tree is a flowchart made of three kinds of parts. The root node is where every example starts, and it asks the single most useful question first. An internal (decision) node asks a question about one feature — "What is the Outlook?" — and has one outgoing branch for each possible answer. A leaf node makes no further test; it simply announces a prediction, because every training example that reached it agrees (or nearly agrees) on the outcome. To classify a brand-new day, you start at the root, answer each question truthfully about that day's weather, follow the matching branch down, and read off the leaf's prediction. There is no other computation involved at prediction time — which is exactly why decision trees are called interpretable: you can print the whole decision procedure and a human can follow it by eye, unlike, say, the millions of weighted connections inside a neural network.

The open question is: out of Outlook, Temperature, Humidity, and Wind, which one should the root ask about first? Guessing is not good enough for a subject this precise — we need a number that measures "how useful is this question," and that number is entropy.

Measuring Uncertainty: Entropy

Think about the 14-day dataset before you know anything about the weather. Someone tells you a random day was picked from it and asks you to guess whether practice happened. Nine times out of 14 the answer is "Yes," so you would reasonably guess "Yes," but you would still be wrong 5 times out of 14. There is genuine uncertainty here. If instead the split had been 14 Yes and 0 No, there would be no uncertainty at all — you would always be right. And if the split had been an even 7 Yes and 7 No, you would be at your most uncertain, since either answer is equally likely.

Entropy is the standard way to turn "how uncertain am I" into a single number. Before writing the formula, we need one small piece of machinery: the base-2 logarithm, written log₂. It simply answers the question "2 raised to what power gives me this number?" So log₂(8) = 3, because 2³ = 8; log₂(1) = 0, because 2⁰ = 1; and log₂ of a fraction less than 1 is negative — for instance log₂(0.5) = −1, because 2⁻¹ = 0.5.

Here is the intuition behind using it for uncertainty. For an outcome that occurs with probability p, the quantity −log₂(p) is large when p is small (a rare, surprising outcome) and equal to 0 when p = 1 (a certain, unsurprising outcome). If you multiply each outcome's "surprise," −log₂(p), by how often that outcome actually happens (its probability p), and add these up over all possible outcomes, you get the average surprise across the whole group. That average is entropy:

H(S) = -Σ p_i · log2(p_i)   (summed over every class i present in S)

For our full 14-day dataset, p(Yes) = 9/14 ≈ 0.643 and p(No) = 5/14 ≈ 0.357. Plugging in:

H(S) = -(0.643 × log2(0.643)) - (0.357 × log2(0.357))
     = -(0.643 × -0.637)      - (0.357 × -1.486)
     = 0.410 + 0.531
     = 0.940 bits

An entropy of 0.940 (out of a maximum possible value of 1.0 for a two-class problem) tells us this dataset starts out fairly uncertain — close to a coin flip, though slightly tilted toward "Yes." That maximum of exactly 1.0 bit occurs only at a perfect 50/50 split; you will see this exact value show up again later in this chapter, for a different subset of the same data, as a useful sanity check.

Information Gain: Choosing the Best Question

Entropy measures uncertainty in one group. Information gain measures how much a particular question would reduce that uncertainty, by comparing the entropy before the split to the weighted-average entropy of the groups it produces:

Gain(S, A) = H(S) - Σ (|S_v| / |S|) · H(S_v)     (summed over each value v that attribute A can take)

The weight |S_v|/|S| matters — a subgroup with only 2 examples should count for less than one with 8, since its entropy is a less reliable measurement of anything.

Let's compute this for Outlook. It splits the 14 days into three groups: Sunny (5 days: 2 Yes, 3 No), Overcast (4 days: 4 Yes, 0 No), and Rain (5 days: 3 Yes, 2 No).

H(Sunny)    = H(2,3) = 0.971
H(Overcast) = H(4,0) = 0.000   (perfectly pure - already all "Yes")
H(Rain)     = H(3,2) = 0.971

Gain(S, Outlook) = 0.940 - [ (5/14)×0.971 + (4/14)×0.000 + (5/14)×0.971 ]
                 = 0.940 - 0.694
                 = 0.246 bits

Running the identical procedure for the other three attributes gives:

Gain(S, Humidity)    = 0.940 - [(7/14)×0.985 + (7/14)×0.592] = 0.152 bits
Gain(S, Wind)         = 0.940 - [(8/14)×0.811 + (6/14)×1.000] = 0.048 bits
Gain(S, Temperature)  = 0.940 - [(4/14)×1.000 + (6/14)×0.918 + (4/14)×0.811] = 0.029 bits

Outlook wins by a wide margin (0.246 bits versus the runner-up's 0.152), so it becomes the root question. Notice something else worth remembering: Humidity's entropy value for the "Normal" subgroup, 0.592, is fairly low even before any further splitting — a hint that Humidity will turn out to be a strong second-level question inside the Sunny branch, which is exactly what happens next.

Building the Full Tree

With Outlook chosen as the root, ID3 (the classic tree-building algorithm that uses information gain) repeats the exact same procedure separately inside each of the three branches, using only the examples that fall into that branch and only the attributes not yet used above it.

Overcast branch (4 examples, all "Yes"): entropy is already 0. There is nothing left to decide — this branch terminates immediately in a leaf that always predicts "Yes." A node becomes a leaf whenever its entropy reaches 0 (every example agrees) or it runs out of attributes to test.

Sunny branch (5 examples: 2 Yes, 3 No, entropy 0.971): we ask which of the three remaining attributes (Temperature, Humidity, Wind) best splits this group. Looking at the five Sunny rows (D1, D2, D8, D9, D11), Humidity happens to split them perfectly: both "High" rows (D1, D2, D8) are "No," and both "Normal" rows (D9, D11) are "Yes." A split with zero remaining entropy achieves a gain of 0.971 − 0 = 0.971 — the theoretical maximum possible at this node, since gain can never exceed the parent's own entropy. No other attribute could possibly beat that, so we don't even need to check Temperature or Wind here; Humidity is guaranteed to win.

Rain branch (5 examples: 3 Yes, 2 No, entropy 0.971): by the identical argument, checking the five Rain rows (D4, D5, D6, D10, D14) against Wind shows a perfectly pure split: all three "Weak" rows (D4, D5, D10) are "Yes," and both "Strong" rows (D6, D14) are "No." Again, a gain of 0.971 is the maximum achievable, so Wind wins this branch without needing to compare against Temperature.

Assembling all of this gives the complete tree:

Read the tree by tracing branches: a Sunny, High-humidity day always says "No"; a Sunny, Normal-humidity day says "Yes"; every Overcast day says "Yes" regardless of anything else; and on a Rain day, a Weak wind says "Yes" while a Strong wind says "No." Every one of the 14 training rows is classified correctly by this five-leaf tree — you can check any row from the table against it.

Gini Impurity: CART's Alternative Yardstick

ID3 (and its refinement C4.5) use entropy, but the algorithm underlying most modern libraries, including scikit-learn's default settings, is called CART (Classification and Regression Trees), and it more commonly uses a different impurity measure called Gini impurity:

Gini(S) = 1 - Σ p_i²

Here's the intuition: p_i² is the probability that if you drew two examples from S at random (with replacement), both happened to belong to class i. Summing p_i² over every class gives the probability that a random pair matches in class; subtracting from 1 gives the probability that a random pair would disagree. Gini impurity is literally "how often would two random members of this group have different labels" — 0 means the group is pure (any two members always agree), and it is largest when classes are evenly mixed.

For our root: Gini(S) = 1 − (0.643² + 0.357²) = 1 − 0.541 = 0.459. Splitting on Outlook gives Gini(Sunny) = 0.480, Gini(Overcast) = 0.000, Gini(Rain) = 0.480, for a weighted-average Gini decrease of 0.459 − [(5/14)(0.480) + (4/14)(0) + (5/14)(0.480)] = 0.459 − 0.343 = 0.116. Doing the same for Humidity gives a Gini decrease of only 0.092. Gini impurity and entropy therefore agree on Outlook as the best root question here, even though they don't produce the same numeric values — 0.116 versus 0.246 bits are simply different units measuring related but distinct notions of impurity. This agreement is common but not universal: on other datasets, the two criteria occasionally rank two closely-matched attributes in a different order, since Gini penalizes impurity slightly differently at the extremes (it doesn't use logarithms at all, which is also why libraries default to it — it is cheaper to compute at scale).

A Common Misconception: "The Tree Finds the Best Overall Tree"

It is tempting to assume that because a decision tree gets every training example right, it must be the best possible tree for the data. This is false, and the reason matters. ID3 and CART are greedy algorithms: at every node, they choose whichever single split has the highest gain right now, and then they never reconsider that choice, no matter what happens deeper in the tree. It is entirely possible for an attribute that looks mediocre at the root to actually set up a much smaller, cleaner tree two levels down — but a greedy algorithm will never discover that combination, because it commits to the locally-best option immediately and never backtracks to compare whole trees against each other. Finding the provably smallest tree consistent with a dataset is a computationally very hard search problem (it belongs to a class of problems where no known algorithm can solve every case quickly as the dataset grows), so in practice essentially every tree-building algorithm you will encounter — ID3, C4.5, CART, and hence scikit-learn — is this kind of greedy, myopic heuristic, not a global optimizer. It usually works well, but "usually well" and "provably optimal" are different claims, and conflating them is the misconception to watch for.

Why a Single Tree Is Unstable

A tree grown all the way down — splitting every node until each leaf is perfectly pure, the way we did above — will always achieve 100% accuracy on its own training data, because it is, in effect, allowed to memorize. That is exactly the danger: a fully-grown tree doesn't just learn the real pattern connecting weather to practice, it also learns to explain away every quirk and coincidence specific to these particular 14 days. If even one or two rows in the training data had been slightly different, some split's information gain could have shifted enough to change which attribute wins at that node — and once an early decision changes, every branch beneath it can end up looking completely different. This sensitivity to small changes in the training data is what statisticians call high variance: the same learning procedure, run on two slightly different samples of data, can output two very different trees. It is the central weakness of decision trees, and it is the entire reason the second half of this chapter exists.

In practice, single trees are made less unstable by constraining their growth — for instance stopping once a node has fewer than some minimum number of examples (min_samples_leaf), or capping how many questions deep the tree is allowed to go (max_depth). These help, but they trade away some accuracy on the training data to gain stability. Bagging, covered next, is a different and often more effective fix: instead of constraining one tree, grow many unconstrained trees and combine them.

Bagging: Bootstrap Aggregating

"Bagging" is short for Bootstrap Aggregating. A bootstrap sample of a training set with n rows is built by drawing n rows one at a time, with replacement — meaning after each draw, the row goes back into the pool and could be picked again. Because of replacement, a typical bootstrap sample contains some of the original rows more than once and leaves other original rows out entirely. For our 14-row dataset, one possible bootstrap draw (using the day numbers) might look like:

Original rows:   D1  D2  D3  D4  D5  D6  D7  D8  D9  D10 D11 D12 D13 D14
Bootstrap draw:  D3, D9, D3, D1, D7, D12, D3, D6, D10, D2, D5, D8, D14, D6

Notice D3 appears three times, D6 appears twice, and D4, D11, and D13 don't appear at all in this particular draw. Grow one full, unconstrained tree on this bootstrap sample using the exact same information-gain procedure from before, and you get a tree that is similar to — but not identical to — the one we built on the original data, because the row counts feeding each split are different. Repeat this B times (a typical forest might use B = 100 to 500), producing B trees, each grown on its own independent bootstrap sample. To make a prediction for a new day, ask every one of the B trees and combine their answers: for classification, take the majority vote; for a numeric target, average the predictions.

The trees disagree with each other precisely because each one overfit its own bootstrap sample's particular quirks — but those quirks are different and largely uncorrelated from tree to tree, while the genuine Outlook/Humidity/Wind signal is present in every bootstrap sample (just with slightly different counts) and so reinforces itself across trees. Averaging tends to cancel the former and preserve the latter. We will make this "tends to cancel" claim mathematically exact soon.

Out-of-Bag Samples: A Free Validation Set

Because bootstrap sampling draws with replacement, a decent fraction of the original rows are left out of any one bootstrap sample entirely — and since that particular tree never saw those rows during training, testing the tree on them gives a fair, honest performance check, without needing to set aside a separate validation set in advance. Rows left out of a given tree's training sample are called out-of-bag (OOB) for that tree.

How large is that "decent fraction"? For one row and one draw, the probability it is not picked is (1 − 1/n), since there are n equally likely rows and only 1 is excluded. Since the n draws are independent, the probability that same row is missed by every one of the n draws is (1 − 1/n) raised to the power n. You don't need calculus to see where this settles — just compute it for growing values of n:

n       (1 - 1/n)^n
1       0.000
5       0.328
10      0.349
50      0.364
100     0.366
1,000   0.368
10,000  0.368

The numbers are visibly settling down around 0.368 as n grows, and they barely move at all between n = 1,000 and n = 10,000. So for any reasonably large dataset, roughly 37% of the original rows are typically left out of a given bootstrap sample (meaning about 63% of rows appear, some of them more than once). In Class 12 calculus, you will meet a special constant called Euler's number, e ≈ 2.71828, and be able to prove this sequence converges to exactly 1/e ≈ 0.3679 — but you don't need that proof to use the fact right now. Every tree in a bagged ensemble comes with roughly a third of the training data as a free, honest test set that it never trained on.

Why Averaging Actually Helps: Variance, Covariance, and Correlation, From Scratch

"Averaging cancels out noise" has been stated twice now as an intuition. Let's build the actual mathematical machinery underneath it, piece by piece, using concrete numbers — because the claim turns out to be only partly true, and knowing exactly where it breaks down is what motivates the random forest's specific fix, coming right after this section.

Step 1 — define variance concretely. Suppose, instead of the Yes/No practice question, we're using bagged trees to predict something numeric: the price, in lakhs of rupees, of a specific 2BHK flat. Five trees, each grown on its own bootstrap sample, predict the same flat's price as: 45, 52, 38, 49, and 61 lakh.

Predictions:        45,  52,  38,  49,  61
Mean:               (45+52+38+49+61)/5 = 49
Deviation from mean: -4,   3, -11,   0,  12
Squared deviation:   16,   9, 121,   0, 144
Average of squares:  (16+9+121+0+144)/5 = 58

That final number, 58, is the variance of these five predictions — precisely defined as the average of the squared distances from the mean. It is large when predictions are spread out and small (down to 0) when every tree agrees exactly. Its square root, √58 ≈ 7.6 lakh, is the standard deviation, which is in the same units (lakhs) as the original predictions and easier to interpret directly: a "typical" tree in this group misses the group average by roughly ₹7.6 lakh.

Step 2 — define covariance and correlation concretely. Now follow two specific trees, Tree 1 and Tree 2, across five different flats (A through E), not five predictions of one flat:

Flat:        A    B    C    D    E
Tree 1:      45   52   38   49   61     (mean = 49)
Tree 2:      42   55   35   50   65     (mean = 49.4)

Dev. Tree1:  -4    3  -11    0   12
Dev. Tree2: -7.4  5.6 -14.4 0.6 15.6
Product:    29.6 16.8 158.4  0  187.2

Look at the signs of the two deviation rows: on every single flat, Tree 1 and Tree 2's deviations share the same sign — both undershoot on Flat C, both overshoot on Flat E. That pattern, "when one is above average the other tends to be above average too," is exactly what it means for two trees to be correlated, and we can measure it precisely by averaging the row of products: (29.6+16.8+158.4+0+187.2)/5 = 78.4. This is the covariance of Tree 1 and Tree 2, written Cov(T1, T2) = 78.4. It's positive because the trees mostly move together; it would be negative if they tended to move in opposite directions, and close to 0 if their errors had no relationship at all.

Covariance's units (lakh², awkwardly) depend on the scale of the data, so we standardize it into correlation, written ρ (rho), by dividing by both standard deviations: ρ = Cov(T1,T2) / (σ₁·σ₂). Computing Tree 2's own variance the same way as Step 1 gives 107.44, so σ₂ = √107.44 ≈ 10.37, and σ₁ = √58 ≈ 7.62. That gives ρ = 78.4/(7.62×10.37) ≈ 0.99. Correlation always sits between −1 and +1, and 0.99 is about as high as it gets: these two trees behave almost identically across all five flats, most likely because both happened to split on the single most dominant feature (say, locality) very early, regardless of which bootstrap sample they were trained on — bootstrap re-sampling barely diversified them at all.

Step 3 — does averaging two trees reduce variance? Derive it. Write each tree's prediction as some shared center μ plus a deviation: T1 = μ + d1, T2 = μ + d2. The average is M = (T1+T2)/2, so:

M - μ = (d1 + d2) / 2

Square both sides using the familiar identity (a+b)² = a² + 2ab + b²:

(M - μ)² = (d1² + 2·d1·d2 + d2²) / 4

Now imagine repeating this whole scenario many times over — many different pairs of bootstrapped trees predicting many different flats — and averaging (M − μ)² across all those repetitions. By the Step 1 definition, that average is Var(M). The average of d1² is Var(T1); the average of d2² is Var(T2); and the average of d1·d2 is exactly Cov(T1,T2), by the Step 2 definition. So:

Var(M) = [Var(T1) + Var(T2) + 2·Cov(T1,T2)] / 4

If both trees share the same variance σ² and their covariance is ρσ² (rearranging the correlation definition, since σ₁=σ₂=σ here):

Var(M) = [σ² + σ² + 2ρσ²] / 4 = σ²(1+ρ) / 2

Check the extremes. If ρ=1 (our Tree 1/Tree 2 pair is nearly this case), Var(M) = σ²(2)/2 = σ² — exactly the same variance as a single tree. Averaging two carbon-copy trees buys you nothing, which matches plain intuition: averaging a number with itself changes nothing. If instead ρ=0 (completely independent trees), Var(M) = σ²/2 — variance is cut in half.

Step 4 — generalize to B trees. Repeating the same expansion — square a sum of B deviations instead of 2 — produces B variance terms and B(B−1) covariance cross-terms. If every tree shares variance σ² and every pair shares the same average correlation ρ, this works out to:

Var(average of B trees) = σ²/B + [(B-1)/B]·ρ·σ²

Sanity-check against Step 3 by setting B=2: σ²/2 + (1/2)ρσ² = σ²(1+ρ)/2 — matches exactly what we derived by hand.

Now plug in real numbers. Our flat-price trees had σ²=58. Suppose a bagged forest of B=100 trees has a typical pairwise correlation of ρ=0.5, a realistic figure for bagged trees that all get to see every feature:

Var(average of 100) = 58/100 + (99/100)·0.5·58 = 0.58 + 28.71 = 29.29

A single tree had variance 58; 100 bagged trees only bring it down to about 29.3 — better than half, but far short of what growing 100 trees might have promised. As B keeps growing, the first term (58/B) keeps shrinking toward 0, but the second term settles at ρσ² = 0.5×58 = 29 and stays there — that floor is set entirely by correlation, and no number of additional trees can push the ensemble's variance below it. This is the precise, quantitative reason plain bagging alone has diminishing returns.

From Bagging to Random Forest: Deliberately Decorrelating the Trees

The formula above pins down exactly what a random forest needs to fix: it needs to lower ρ, not just grow more trees. A random forest is bagging plus one additional rule: at every single split in every tree — not just once at the root — the algorithm is only allowed to consider a random subset of the available features (a common default is √p features out of p total, for classification; roughly p/3 for regression), and it must pick the best split from within that random subset alone.

Here is why that specific change works. In plain bagging, if one feature (Outlook, in our example; locality, in the flat-price example) is dramatically more predictive than the others, nearly every bootstrapped tree will pick it as the very first split anyway, almost regardless of which bootstrap sample it happened to see — because it wins the information-gain contest by such a wide margin. That is exactly the mechanism that produced our Tree 1/Tree 2 pair's ρ≈0.99: both trees leaned on the same dominant feature immediately, so bootstrap sampling's randomness barely mattered. By randomly hiding that dominant feature from many of the splits, a random forest forces some trees to rely on the second- or third-best feature instead at various points, producing genuinely different tree shapes — a lower ρ.

Plug the same numbers back in, but now suppose feature-subsampling has pulled the typical pairwise correlation down to ρ=0.2:

Var(average of 100) = 58/100 + (99/100)·0.2·58 = 0.58 + 11.48 = 12.06

Bagging's floor, at ρ=0.5, was ρσ²=29; random forest's floor, at ρ=0.2, is ρσ²=11.6 — less than half. Crucially, random forest doesn't shrink each individual tree's own σ² (a tree denied its favorite feature at some splits may, if anything, be slightly noisier on its own) — the entire gain comes from pulling ρ down, and the formula above is precisely why that single change is disproportionately effective. There is a real trade-off buried here too: restricting each split to very few candidate features can weaken individual trees enough (raising their own σ²) to outweigh the correlation benefit, which is why max_features is a hyperparameter worth tuning rather than a fixed universal constant.

One more practical payoff of the impurity-decrease bookkeeping we've been doing by hand: a trained random forest can report, for every feature, how much total impurity decrease (summed across every split, in every tree, where that feature was used, weighted by how many examples reached that split) it was responsible for. Normalized to add up to 1, this is feature importance — a ranked list of which inputs the forest actually leaned on, entirely automatic, no extra computation beyond what tree-building already does.

Building It in Code

Here is the same 14-row dataset built with scikit-learn, comparing a single decision tree against a random forest. Both were actually run to produce the output shown below — nothing here is a guess.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.ensemble import RandomForestClassifier

data = {
    "outlook":     ["Sunny","Sunny","Overcast","Rain","Rain","Rain","Overcast",
                     "Sunny","Sunny","Rain","Sunny","Overcast","Overcast","Rain"],
    "temperature": ["Hot","Hot","Hot","Mild","Cool","Cool","Cool",
                     "Mild","Cool","Mild","Mild","Mild","Hot","Mild"],
    "humidity":    ["High","High","High","High","Normal","Normal","Normal",
                     "High","Normal","Normal","Normal","High","Normal","High"],
    "wind":        ["Weak","Strong","Weak","Weak","Weak","Strong","Strong",
                     "Weak","Weak","Weak","Strong","Strong","Weak","Strong"],
    "play":        ["No","No","Yes","Yes","Yes","No","Yes",
                     "No","Yes","Yes","Yes","Yes","Yes","No"]
}
df = pd.DataFrame(data)
X = pd.get_dummies(df[["outlook","temperature","humidity","wind"]])
y = df["play"]

tree = DecisionTreeClassifier(criterion="entropy", random_state=0)
tree.fit(X, y)
print(tree.get_depth(), tree.get_n_leaves(), tree.score(X, y))
print(export_text(tree, feature_names=list(X.columns)))

Output:
4 7 1.0

|--- outlook_Overcast <= 0.50
|   |--- humidity_Normal <= 0.50
|   |   |--- outlook_Rain <= 0.50
|   |   |   |--- class: No
|   |   |--- outlook_Rain >  0.50
|   |   |   |--- wind_Strong <= 0.50
|   |   |   |   |--- class: Yes
|   |   |   |--- wind_Strong >  0.50
|   |   |   |   |--- class: No
|   |--- humidity_Normal >  0.50
|   |   |--- wind_Weak <= 0.50
|   |   |   |--- temperature_Cool <= 0.50
|   |   |   |   |--- class: Yes
|   |   |   |--- temperature_Cool >  0.50
|   |   |   |   |--- class: No
|   |   |--- wind_Weak >  0.50
|   |   |   |--- class: Yes
|--- outlook_Overcast >  0.50
|   |--- class: Yes

Reported depth is 4 and there are 7 leaves — deeper and leafier than the 5-leaf, depth-2 tree we built by hand. This is not a mistake, and it's a genuinely important distinction: scikit-learn's CART implementation only ever makes binary splits, even on a one-hot-encoded categorical feature. Where ID3 let Outlook split three ways at once (Sunny / Overcast / Rain, all from one node), CART has to peel it off one category at a time — first asking "is outlook_Overcast > 0.5?", and only later, deeper in the tree, separately asking about outlook_Rain. Trace the printed rules and they still encode identical logic to our hand-built tree: outlook_Overcast > 0.5 alone gives "Yes" (our Overcast leaf); everything else eventually asks about humidity_Normal, matching the Sunny branch; and wind_Strong/wind_Weak checks reproduce the Rain branch. Same decisions, more binary questions to express them.

forest = RandomForestClassifier(n_estimators=200, max_features="sqrt", random_state=0)
forest.fit(X, y)
importances = pd.Series(forest.feature_importances_, index=X.columns)
print(importances.sort_values(ascending=False).head(5))

Output:
outlook_Sunny       0.157
outlook_Overcast    0.152
outlook_Rain        0.126
humidity_High       0.122
wind_Weak           0.109

With this particular random seed, the three Outlook dummy columns do occupy the top three importance slots, consistent with Outlook having by far the largest information gain of any single attribute. But that exact ranking is not a guaranteed outcome — running the identical code with a few other random seeds shows humidity_Normal or wind_Weak occasionally displacing one of the Outlook columns from the top three, because humidity_Normal is also a genuinely strong single-split discriminator on its own (recall its entropy was only 0.592, quite pure already), and both the bootstrap sampling and the random feature subsampling add run-to-run randomness that a dataset this tiny cannot fully average away. On a real dataset with thousands of rows, feature-importance rankings stabilize far more; on 14 rows, treat the exact order as indicative, not guaranteed.

Where This Fits in Your Exams

CBSE's Artificial Intelligence elective (Subject Code 417), offered in Classes IX and X, includes classification-based machine learning as a topic, and decision trees are the standard concrete example used there. This chapter is meant to sit underneath that syllabus coverage: rather than only being able to name "entropy" and "information gain," you can now compute both by hand for a real dataset and explain exactly why a tree makes each choice it makes — which is precisely the kind of reasoning board-exam application questions tend to ask for, once you move past pure definitions.

Looking further ahead, the toolkit you built in this chapter — probability, logarithms used to measure uncertainty, and now variance, covariance, and correlation computed from real numbers — is exactly the material formalized in Class 11 and 12 statistics and probability chapters, and it resurfaces again if you later pursue engineering entrance preparation (JEE, BITSAT) or a computer science stream. Treat today's numeric derivations as an early, hands-on rehearsal of ideas you will meet again in more abstract form, not as isolated facts to memorize for their own sake.

Check Yourself

• Using the counts given for Temperature (Hot: 2 Yes/2 No; Mild: 4 Yes/2 No; Cool: 3 Yes/1 No), verify by hand that Gain(S, Temperature) ≈ 0.029, and explain in one sentence why this makes Temperature the least useful root-level question of the four attributes.
• A tree grown with min_samples_leaf=1 (allowed to keep splitting until every leaf has just one example) will reach 100% training accuracy on almost any dataset with no contradictory duplicate rows. Why does that fact alone make training accuracy a poor way to judge whether a tree is actually a good model?
• If you built B=200 trees but trained every single one on the exact same bootstrap sample (no re-sampling at all), what value would ρ equal, and what would the variance formula from this chapter then predict for Var(average)? What does this tell you about why re-sampling — not just "many trees" — is essential?
• Using Var(average of B) = σ²/B + [(B−1)/B]·ρσ², explain why increasing B alone can never fully eliminate an ensemble's variance when ρ > 0, and identify exactly which term in the formula is responsible.
• A specific row appeared 3 times in one tree's bootstrap sample and 0 times in another tree's bootstrap sample. Which of these two trees can legitimately use that row as an out-of-bag test case, and why can't the other one?

Summary

• A decision tree recursively asks the single most useful yes/no or multi-way question at each node, chosen by whichever attribute yields the highest information gain (entropy-based) or Gini decrease, and stops at leaves once a group is pure or no attributes remain.
• Entropy H(S) = −Σp_ilog₂(p_i) measures uncertainty in a group; information gain is the entropy removed by a split, weighted by group sizes.
• Gini impurity, 1−Σp_i², is CART's cheaper alternative to entropy and usually — not always — agrees on which split is best.
• Tree-building algorithms are greedy: locally optimal at each node, with no guarantee of a globally optimal tree.
• Fully-grown single trees overfit and are high-variance: small changes in training data can produce very different trees.
• Bagging grows many trees on independent bootstrap samples (drawn with replacement) and combines them by voting or averaging; roughly 37% of rows are left "out-of-bag" per tree, giving a free validation check.
• Averaging B predictions with per-tree variance σ² and pairwise correlation ρ has variance σ²/B + [(B−1)/B]ρσ² — correlation, not tree count, sets the floor on how much bagging can help.
• Random forests add random feature subsampling at every split specifically to lower ρ, pushing that floor down further than plain bagging can reach, at the cost of slightly weaker individual trees.
• Feature importance in a forest is the total impurity decrease a feature is responsible for across every split in every tree, and is computed for free as a byproduct of training.