Meera and Arjun spent three weeks of their summer break building a machine learning model that predicts flat prices in their city, for a school hackathon on "AI for Real Estate." Their model was good — on flats it had never seen, its price guesses were off by less than 5% on average. At the judging round, a judge from a housing-finance company asked one question: "Your model says this particular flat is worth ₹79 lakh. Why? Which features pushed the price up, and by how much?" Meera and Arjun looked at each other. Their model was a random forest with 200 trees voting internally. They had no idea how to answer, beyond "the algorithm decided." They lost marks — not because the model was inaccurate, but because it could not explain itself.
This is the exact gap that this chapter fixes. A model that predicts well but cannot say why is unusable in situations that matter — a bank cannot reject your loan application and legally tell you only "the algorithm said no" (the Reserve Bank of India's fair-lending guidance requires lenders to be able to explain adverse credit decisions to the applicant), and a hackathon judge, quite reasonably, will not give full marks to a black box. You are going to learn two concrete, computable techniques that open the box: SHAP (SHapley Additive exPlanations), which explains one single prediction by fairly splitting credit among the features, and permutation importance, which ranks features by how much the model's overall accuracy depends on each one. By the end, you will be able to compute both by hand on a small example, and read real output from Python's shap library correctly.
Two Different Questions: "What Matters Overall?" vs. "Why This Answer?"
Before building any technique, separate two questions that people often blur together.
- Global importance: "Across all flats in my dataset, which feature does my model rely on the most, on average?" This is a single ranked list — one importance score per feature, computed once.
- Local explanation: "For this one specific flat, why did the model output ₹79 lakh instead of the city's average price of ₹60 lakh?" This is computed fresh for every individual prediction, and the answer can differ from house to house — a huge floor number might barely matter for one flat but matter a lot for another, depending on what else is true about that flat.
Permutation importance answers only the first question. SHAP can answer both — it explains individual predictions, and you can average those individual explanations to get a global ranking too. That is one reason SHAP has become the default explainability tool in serious machine learning work: one method, two kinds of answers.
A First, Simpler Idea: Splitting Feature Importance From a Decision Tree
Before SHAP, it helps to see the crude tool it replaced. If your model is a single decision tree, you already have a rough importance measure for free: every time the tree splits on a feature, that split reduces "impurity" (roughly, how mixed-up the target values are in that branch) by some amount. Add up the impurity reduction credited to each feature across every split in the tree, and you get that feature's built-in importance score.
Say a small tree predicting whether a loan is approved splits three times: once on "credit score" (impurity drops by 0.30), once on "monthly income" (impurity drops by 0.12), and once on "existing EMI count" (impurity drops by 0.08). Total impurity reduction is 0.50. Scikit-learn reports these as fractions of the total, so credit score gets 0.30/0.50 = 60%, income gets 24%, EMI count gets 16%. That is a real, useful number, and it is what you get for free from model.feature_importances_ in scikit-learn.
But it has two real weaknesses. First, it is a global number only — it cannot tell you why one particular applicant was rejected. Second, it is tied to how the tree happened to split; a feature that is genuinely important but appears lower down in the tree (because a correlated feature got split on first) gets under-credited. You need something that does not depend on the accident of which feature the tree algorithm split on first. That is exactly the gap Shapley values were built to close — nearly seventy years before anyone used them for machine learning.
A Fair Way to Split Credit: The Shapley Value Idea
In 1953, the mathematician Lloyd Shapley was working on a completely different problem in game theory: if a group of people cooperate to produce some total value — say, a project team that jointly earns a bonus — how do you split the bonus fairly among them, given that each person's individual contribution depends on who else is already in the group? Shapley's answer, the Shapley value, is computed by imagining every possible order in which people could join the group one at a time, measuring each person's marginal contribution at the moment they join in each order, and then averaging that contribution across all the orders. This idea, developed decades before anyone applied it to AI, later became part of the work for which Shapley shared the 2012 Nobel Memorial Prize in Economic Sciences (for the theory of stable allocations and market design, building on the same cooperative-game-theory foundations as the Shapley value).
Applied to machine learning, the "group of cooperating people" becomes the group of input features, and the "bonus being split" is the gap between one specific prediction and the model's average prediction. This is the whole idea behind SHAP: treat each feature as a player who "joins" the prediction one at a time, in every possible order, measure how much the predicted price moves each time a feature joins, and average those movements across every order. What comes out is a fair, mathematically guaranteed way to split "why this house is worth ₹79 lakh instead of the average ₹60 lakh" among Area, Metro distance, and Floor.
The Full Worked Example: Pricing One Flat
Here is the exact flat Meera and Arjun's model was asked about. It has three features: Area (it is a large 1,450 sq. ft. flat), Metro distance (only 0.5 km from the nearest metro station), and Floor (it is on the 12th floor, a little higher than typical). Across their whole dataset, the average flat price is ₹60 lakh — call this the base value. For this specific flat, the model predicts ₹79 lakh. Shapley values will explain exactly how that extra ₹19 lakh gets divided among the three features.
To do this we need a "value function," written v(S), that answers: "if we only reveal the features in the set S to the model and average out the rest, what price would it estimate?" Working with Meera and Arjun's trained model on this specific flat gives these eight values (there are eight because three features have 2³ = 8 possible subsets, including the empty set and the full set):
- v(∅) = ₹60L — no features revealed; just the dataset average
- v({Area}) = ₹65L — only told it is 1,450 sq ft
- v({Metro}) = ₹68L — only told it is 0.5 km from the metro
- v({Floor}) = ₹61L — only told it is on the 12th floor
- v({Area, Metro}) = ₹75L
- v({Area, Floor}) = ₹71L
- v({Metro, Floor}) = ₹64L
- v({Area, Metro, Floor}) = ₹79L — everything revealed; the actual prediction
With three features there are 3! = 6 possible orders in which they could "join" the prediction. For each order, we compute each feature's marginal contribution — how much the estimate jumps when that feature is added, given what was already known. Let A, M, F stand for Area, Metro, Floor.
- Order A → M → F: Area joins first: 65 − 60 = 5. Metro joins next (Area already known): 75 − 65 = 10. Floor joins last: 79 − 75 = 4.
- Order A → F → M: Area first: 65 − 60 = 5. Floor next: 71 − 65 = 6. Metro last: 79 − 71 = 8.
- Order M → A → F: Metro first: 68 − 60 = 8. Area next: 75 − 68 = 7. Floor last: 79 − 75 = 4.
- Order M → F → A: Metro first: 68 − 60 = 8. Floor next: 64 − 68 = −4. Area last: 79 − 64 = 15.
- Order F → A → M: Floor first: 61 − 60 = 1. Area next: 71 − 61 = 10. Metro last: 79 − 71 = 8.
- Order F → M → A: Floor first: 61 − 60 = 1. Metro next: 64 − 61 = 3. Area last: 79 − 64 = 15.
Notice every order's three contributions add up to 19 (for example, 5 + 10 + 4 = 19, and 8 + (−4) + 15 = 19) — that is not a coincidence, it is simply the base value plus everything the three features contribute always reaching the full prediction, whatever order you add them in. Now average each feature's contribution across all six orders:
- Area: (5 + 5 + 7 + 15 + 10 + 15) / 6 = 57 / 6 = 9.5
- Metro: (10 + 8 + 8 + 8 + 8 + 3) / 6 = 45 / 6 = 7.5
- Floor: (4 + 6 + 4 − 4 + 1 + 1) / 6 = 12 / 6 = 2.0
Check: 9.5 + 7.5 + 2.0 = 19, and 60 + 19 = 79 — exactly the model's prediction. This always-true property is called the efficiency property of Shapley values: the base value plus every feature's SHAP value must add up to exactly the actual prediction, with nothing left over and nothing double-counted. It is what makes a SHAP explanation trustworthy as arithmetic, not just a plausible-looking story.
Common misconception, and why it matters: a natural guess is that whichever feature has the highest value alone — v({feature}) by itself — must end up with the highest Shapley value. Look again at the numbers above: Metro alone gives ₹68L (a jump of 8 over the base), while Area alone gives only ₹65L (a jump of 5). By that solo measure, Metro looks more important than Area. Yet Area's final Shapley value (9.5) is larger than Metro's (7.5). What happened? Area's contribution grows a lot once Metro is already known (orders 3, 4, 6 above show Area contributing 7, 15, and 15 respectively when added after Metro) — the two features interact, and a large flat close to the metro is worth disproportionately more together than either fact suggests alone. A method that only looked at solo effects would get the ranking backwards. This is exactly why Shapley values average over every possible order instead of picking one: no single order can be trusted to reveal a feature's true, fair contribution when features interact with each other, and in real datasets, they almost always do.
Seeing It as a Picture: The SHAP Waterfall
The numbers above are usually drawn as a waterfall chart: start at the base value, then let each feature push the bar up (or down, if its SHAP value were negative) until you land exactly on the final prediction.
Read left to right: the gray bar is where every flat starts (₹60L, the dataset average, before the model knows anything about this specific flat). The green bars are the Shapley values you just calculated by hand, stacked one after another — Area pushes the running total from ₹60L to ₹69.5L, Metro pushes it from ₹69.5L to ₹77L, and Floor pushes it the final bit to ₹79L. The navy bar at the bottom is simply the sum of everything above it, landing exactly on the model's actual output. This is the chart Meera and Arjun should have shown the judge.
From One Prediction to the Whole Dataset: Global SHAP Importance
SHAP values are computed per prediction, per feature — one flat gives you three numbers (9.5, 7.5, 2.0 here), and a dataset of 500 flats gives you 500 rows of three numbers each. To turn this into a single global importance ranking (answering "which feature matters most overall?"), take the average of the absolute value of each feature's SHAP value across every row in the dataset. You use the absolute value because a feature that sometimes pushes the price up by a lot and sometimes pushes it down by a lot is clearly influential, even though its raw contributions might average out close to zero. A feature whose SHAP value hovers near zero for almost every flat, on the other hand, genuinely is not doing much work in the model. This is how SHAP gives you both the "why this one prediction" story and the "what matters overall" ranking from the exact same underlying computation — unlike the tree-impurity importance from earlier, which only ever gave you the global number.
A Second, Independent Check: Permutation Importance
SHAP is powerful but computationally heavier, especially for models with many features. A cheaper, purely global technique that Meera and Arjun's team also tried is permutation importance. The idea: if a feature genuinely matters to the model, then scrambling its values should hurt the model's accuracy a lot. If a feature barely matters, scrambling it should barely change anything.
The algorithm, concretely:
- Train the model once, and measure its baseline accuracy on a held-out validation set. Do not retrain again during this whole process.
- Pick one feature column. Randomly shuffle (permute) its values up and down the rows, so each row now has a random, mismatched value for that one feature while every other column stays untouched.
- Feed this shuffled dataset through the already-trained model and measure accuracy again.
- The drop in accuracy — baseline minus shuffled — is that feature's permutation importance. Undo the shuffle, and repeat for the next feature.
Suppose a different model from the same hackathon team — a loan-approval classifier using Credit Score, Monthly Income, and City Tier — scores 90% baseline accuracy on the validation set. Shuffling Credit Score alone drops accuracy to 65% — a fall of 25 percentage points, telling you the model leans heavily on this one column; scramble it and the model is barely better than guessing on many cases. Shuffling City Tier alone drops accuracy to 82.5% — a much smaller fall of 7.5 percentage points, meaning City Tier helps, but the model does not depend on it nearly as much. Ranked by permutation importance: Credit Score (−25 points) is far more important than City Tier (−7.5 points).
Permutation importance is quick to compute and does not require understanding the model's internals at all — it treats the model as a black box and only checks input-output behaviour, which is why it works identically for a decision tree, a random forest, or a neural network. Its weakness is that it is global-only (no per-prediction story) and it can mislead you when features are correlated: if Credit Score and Monthly Income tend to move together in real applicants, shuffling only Credit Score creates rows with unrealistic, contradictory combinations (a very low credit score paired with a very high income that never actually occurs together), and the model's behaviour on such invented rows may not reflect anything meaningful about the real world. SHAP, by contrast, is built to handle exactly this kind of feature interaction correctly, which is one reason it has become the more trusted tool for high-stakes decisions like loan approvals.
Trying It in Code
First, here is the hand calculation from the flat-price example, written as a short Python program using every ordering explicitly, so you can check the arithmetic yourself:
from itertools import permutations
def v(S):
values = {
frozenset(): 60,
frozenset({'Area'}): 65,
frozenset({'Metro'}): 68,
frozenset({'Floor'}): 61,
frozenset({'Area', 'Metro'}): 75,
frozenset({'Area', 'Floor'}): 71,
frozenset({'Metro', 'Floor'}): 64,
frozenset({'Area', 'Metro', 'Floor'}): 79,
}
return values[frozenset(S)]
features = ['Area', 'Metro', 'Floor']
contributions = {f: [] for f in features}
for order in permutations(features):
coalition = set()
for f in order:
before = v(coalition)
coalition.add(f)
after = v(coalition)
contributions[f].append(after - before)
for f in features:
shap_value = sum(contributions[f]) / len(contributions[f])
print(f"{f}: {shap_value:.1f}")
total = 60 + sum(sum(contributions[f]) / 6 for f in features)
print(f"Base + all SHAP values = {total:.1f}")
Tracing it: the outer loop visits all six orderings of the three features; for each ordering, the inner loop reveals one feature at a time, looks up the value function before and after, and records the jump. After all six orderings, contributions['Area'] holds the list [5, 5, 7, 15, 10, 15], and dividing its sum (57) by 6 gives 9.5 — matching the hand calculation exactly. The program prints:
Area: 9.5
Metro: 7.5
Floor: 2.0
Base + all SHAP values = 79.0
In real projects you never build the value function or the permutation loop by hand — you use the shap library, which computes exact Shapley values efficiently for tree-based models using an algorithm called TreeExplainer, instead of brute-forcing every ordering (which becomes impossibly slow once you have more than about 15–20 features, since the number of orderings grows factorially):
import shap
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=200, random_state=0)
model.fit(X_train, y_train)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
row = 0
predicted = model.predict(X_test.iloc[[row]])[0]
reconstructed = explainer.expected_value + shap_values[row].sum()
print(f"Model's actual prediction: {predicted:.2f}")
print(f"Base value + summed SHAP values: {reconstructed:.2f}")
The last two lines are a direct, practical use of the efficiency property you verified by hand earlier: explainer.expected_value is the base value (the average prediction over the training set, playing the same role as v(∅) = 60 above), and shap_values[row].sum() is the total of that row's per-feature SHAP values. Adding them should reproduce the model's actual prediction for that row, to within tiny floating-point rounding. If those two printed numbers do not match closely, that is a real signal something is set up wrong — perhaps the wrong explainer for the model type — because the efficiency property is a mathematical guarantee, not an approximation.
Where This Actually Gets Used
You will meet this again well before university. CBSE Class 10 Artificial Intelligence and Class 11–12 Informatics Practices projects are increasingly graded on whether a model's decisions can be explained, not just how accurate it is — a project report that includes a SHAP waterfall for a sample prediction reads very differently to an evaluator than one that just states an accuracy percentage. Beyond school, the Reserve Bank of India's guidelines on algorithmic lending push banks and NBFCs toward being able to justify credit decisions to individual applicants, which in practice means someone on the team has computed exactly the kind of per-prediction explanation you built above. Fraud-detection systems used by UPI apps and card networks face the same requirement in reverse: when a transaction is blocked as suspicious, an analyst reviewing the flag benefits enormously from seeing which specific factors (unusual location, unusual amount, unfamiliar merchant) drove that one decision, rather than a single opaque risk score.
Check Your Understanding
- Using the value function from the worked example (v(∅)=60, v({Metro})=68, v({Area,Metro})=75, v({Area,Metro,Floor})=79), calculate Metro's marginal contribution in the order Floor → Metro → Area, given that v({Metro,Floor}) = 64.
- A model's base value is ₹40L and its SHAP values for a particular house are Area = +6, Location = +9, Age = −3. What is the model's actual prediction for this house, and which property of Shapley values guarantees your answer?
- Using the table of v(S) values from the worked example, compute Metro's marginal contribution when it joins after Area only (v({Area,Metro}) − v({Area})), and compute Area's marginal contribution when it joins after Metro only (v({Area,Metro}) − v({Metro})). Are the two numbers equal? What does this tell you about why Shapley values are averaged across every possible ordering rather than computed from just one order?
- A permutation importance test on a 200-row validation set starts at 84% baseline accuracy. After shuffling feature X, accuracy falls to 71%. After shuffling feature Y, accuracy falls to 80%. Which feature is more important to the model, and by how many percentage points does each one matter?
- A classmate argues: "Feature importance from a decision tree's impurity reduction and SHAP global importance are really the same thing, so it doesn't matter which one you report." Give one concrete reason this claim is wrong.
Answer Key
- Metro joins after Floor, so its marginal contribution is v({Metro,Floor}) − v({Floor}) = 64 − 61 = 3.
- Prediction = base value + sum of SHAP values = 40 + 6 + 9 + (−3) = ₹52L. This is guaranteed by the efficiency property, which states that a prediction always equals the base value plus the sum of all feature SHAP values, with nothing left unaccounted for.
- Metro after Area: 75 − 65 = 10. Area after Metro: 75 − 68 = 7. These are not equal (10 ≠ 7), which shows that a feature's marginal contribution genuinely depends on which other features are already "known" when it joins — features interact. Averaging over all six orderings is exactly how Shapley values give each feature one single, fair number instead of a value that changes depending on which order you happened to pick.
- Feature X is more important: shuffling it costs 84 − 71 = 13 percentage points, versus feature Y's 84 − 80 = 4 percentage points.
- Tree impurity importance is a global-only number tied to the accident of which feature the tree algorithm happened to split on and how high up in the tree; it cannot explain any single prediction and can under-credit a feature that a correlated feature "stole" splits from. SHAP global importance is derived by averaging genuine per-prediction Shapley values (which satisfy the efficiency property and account fairly for feature interactions), and the same underlying numbers can also explain individual predictions — something tree impurity importance can never do.
Summary
A model that predicts accurately but cannot explain a specific answer fails in exactly the situations where explanations matter most — loan decisions, fraud flags, and hackathon judging panels included. Built-in tree importance gives a rough global ranking but cannot explain individual predictions and is distorted by which feature the tree happened to split on first. SHAP fixes both problems by treating features as players in a cooperative game: imagine every possible order in which features could "join" a prediction, measure each feature's marginal contribution to the running estimate in every order, and average across all orders to get one fair Shapley value per feature. These values always satisfy the efficiency property — base value plus every feature's SHAP value reconstructs the exact prediction, as you verified in the ₹60L-to-₹79L flat example and again in the Python trace. Averaging the absolute SHAP values across an entire dataset turns these per-prediction numbers into a genuine global importance ranking too. Permutation importance offers a simpler, model-agnostic global-only alternative: shuffle one feature's values, measure how much accuracy drops, and that drop is the importance score — cheap to compute, but blind to individual predictions and unreliable when features are correlated. Knowing both tools, and knowing which question each one actually answers, is what separates a model you can defend to a judge, a bank customer, or an examiner from one where "the algorithm decided" is the only answer you have.