Causal Inference: Understanding Cause and Effect

Every May in Delhi, air-conditioner showrooms sell far more units than they do in January. In the very same months, the ice-cream cart outside your school also sells far more kulfis and cones. If you only looked at the sales numbers — AC sales going up, ice-cream sales going up, month after month, together — you might write a program that concludes: "Buying an AC causes people to eat more ice-cream." That sentence sounds silly the moment you read it. But this exact mistake — mistaking two things that move together for one thing causing the other — is made every day by people analysing real data, and even by algorithms that recommend products, rank search results, or decide which students get flagged as "at risk." This chapter is about learning to catch that mistake before it happens, using the same careful, step-by-step thinking you already use when you trace through a program.

Causal inference is the branch of data analysis concerned with a very specific question: not just "do X and Y move together?" but "does changing X actually make Y change?" It is the discipline that separates a pattern in a spreadsheet from a lever you can actually pull. If you are ever going to write code that analyses data and draws conclusions from it — which app feature to launch, which UPI reminder increases repayment, which fertiliser increases crop yield — this is the single most important habit of mind you can build.

Two Numbers Moving Together: A Worked Example

Let's build the AC/ice-cream observation into real numbers, because "they seem related" is a feeling, and feelings are exactly what causal inference replaces with arithmetic. Suppose we record, for five months, the average temperature, the number of ACs sold (in thousands, across a city), and the number of ice-cream units sold (in thousands):

• January — Temperature 15°C, AC sales 2, Ice-cream sales 4
• March — Temperature 25°C, AC sales 10, Ice-cream sales 18
• May — Temperature 40°C, AC sales 50, Ice-cream sales 68
• July — Temperature 35°C, AC sales 32, Ice-cream sales 46
• September — Temperature 30°C, AC sales 20, Ice-cream sales 30

To measure whether two lists of numbers move together, statisticians use a quantity called covariance. It looks intimidating in a textbook, but the idea behind it is arithmetic you already know: for each pair of values, check whether both are above their own average at the same time, or both below at the same time, or one above while the other is below. We can build this ourselves, one step at a time.

Step 1: find the mean (average) of each list.

• Mean temperature = (15+25+40+35+30) / 5 = 145 / 5 = 29°C
• Mean AC sales = (2+10+50+32+20) / 5 = 114 / 5 = 22.8
• Mean ice-cream sales = (4+18+68+46+30) / 5 = 166 / 5 = 33.2

Step 2: for each month, find how far each value is from its own mean (its "deviation"). A positive deviation means that month was above average; a negative deviation means below average.

• January: temp −14, AC −20.8, ice-cream −29.2 (all below average)
• March: temp −4, AC −12.8, ice-cream −15.2 (all below average)
• May: temp +11, AC +27.2, ice-cream +34.8 (all above average)
• July: temp +6, AC +9.2, ice-cream +12.8 (all above average)
• September: temp +1, AC −2.8, ice-cream −3.2 (mixed, but close to zero)

Notice the pattern already, without any further calculation: in four out of five months, temperature, AC sales, and ice-cream sales are all above average together, or all below average together. That "moving together" pattern is exactly what covariance measures numerically. Step 3 turns it into a single number: for each month, multiply the two deviations you're comparing, then add up all five products. If most pairs of deviations have the same sign (both positive or both negative), the products are positive and the sum is a large positive number — a strong "moves together" signal. If the signs tend to disagree, the sum comes out negative.

Here is a function that does exactly this — it is a simplified version of covariance, built from nothing but sums and averages:

def comovement(x, y):
    n = len(x)
    mean_x = sum(x) / n
    mean_y = sum(y) / n
    total = 0
    for i in range(n):
        total += (x[i] - mean_x) * (y[i] - mean_y)
    return total

temp      = [15, 25, 40, 35, 30]
ac_sales  = [2, 10, 50, 32, 20]
ice_sales = [4, 18, 68, 46, 30]

print(comovement(temp, ac_sales))       # 694.0
print(comovement(temp, ice_sales))      # 926.0
print(comovement(ac_sales, ice_sales))  # 1875.2

Trace it by hand for temp and ac_sales to check the function is doing what we described: the deviation products are (−14)(−20.8)=291.2, (−4)(−12.8)=51.2, (11)(27.2)=299.2, (6)(9.2)=55.2, (1)(−2.8)=−2.8. Adding these: 291.2 + 51.2 + 299.2 + 55.2 − 2.8 = 694.0 — matching the program's output exactly. Every one of the three pairs above comes out strongly positive. Temperature and AC sales move together (694.0). Temperature and ice-cream sales move together (926.0). And — this is the important part — AC sales and ice-cream sales also move together, just as strongly (1875.2), even though nobody buying an air conditioner has any effect whatsoever on kulfi sales.

Correlation Is Not Causation

What we just computed is correlation: a measurable tendency for two quantities to rise and fall together. Causation is a much stronger claim: that changing one quantity directly produces a change in the other. The AC/ice-cream example proves, with real arithmetic, that correlation can be strong and real while causation is completely absent. Both AC sales and ice-cream sales are driven by a third variable — temperature — that causes them both. Neither one causes the other.

This third variable is called a confounding variable (or lurking variable): a hidden cause that influences both of the variables you're studying, creating a correlation between them that has no direct causal link. Confounding is the single most common reason two correlated things turn out not to be causally related, and trained data analysts are taught to ask, before believing any causal claim: "Is there some third factor that could be driving both of these?"

Reverse Causation: Getting the Arrow Backwards

Confounding is one way correlation misleads you. A second, sneakier way is reverse causation — assuming X causes Y, when actually Y causes X, and you simply guessed the arrow's direction wrong. A classic illustration used in causal-inference courses: cities that employ more police officers tend to also report more crimes. It is tempting to read this as "hiring more police increases crime" — but the far more sensible direction is the reverse: cities with more crime hire more police officers to respond to it. The correlation is real; the arrow just points the opposite way from the careless first guess.

This mistake is easy to make because correlation, as a number, has no direction built into it — comovement(x, y) gives exactly the same value as comovement(y, x), since multiplication doesn't care about order. The data alone cannot tell you which variable is the cause and which is the effect, or whether either one is. Deciding the direction requires reasoning about the situation, not just running a calculation.

Coincidence: When N Is Small

A third trap is plain chance. If you only have a handful of data points, or if you search through hundreds of possible variable pairs looking for anything that lines up, you will eventually find pairs that appear strongly correlated purely by luck, with no underlying connection at all — not a confounder, not a reversed arrow, just coincidence. This is why a single semester's worth of scores for one classroom, or five days of app-usage logs, should never be trusted to prove a causal claim: with so little data, random fluctuation alone can produce a convincing-looking pattern. Real causal inference distrusts small samples and one-off patterns, and asks whether the relationship holds up across many independent observations.

The Misconception, Named and Corrected

Put plainly, the most common error in reasoning about data is this: "If X and Y are correlated, X must cause Y." This is false, and by now you have concrete evidence why — the AC/ice-cream calculation produced a real, strongly positive correlation with zero causal link. A correlation between X and Y is consistent with four different underlying realities, and the data alone cannot tell you which one you're looking at:

• X actually causes Y (the situation you hoped for)
• Y actually causes X (reverse causation)
• Some third variable Z causes both X and Y (confounding)
• The pattern is coincidence, especially likely with a small sample

Whenever you see a causal claim built only on a correlation — in a news headline, an app's analytics dashboard, or your own code's output — run through this checklist before believing it: Could something else be driving both? Could the arrow be reversed? Is my sample even big enough to trust this pattern?

How Do You Actually Prove Causation? The Randomized Experiment

If observing data that already exists (this is called observational data — collected by watching the world as it naturally happens) can't settle causation on its own, what can? The answer, developed over the twentieth century by statisticians and now used constantly in medicine, technology, and agriculture, is the randomized controlled trial (RCT). The idea is simple to state: instead of just observing who already does X and comparing them to who doesn't, you actively and randomly assign some subjects to receive X (the treatment group) and others to not receive it (the control group) — using a coin flip or its equivalent, not anyone's judgement or preference.

Randomization is the key move, and it's worth understanding exactly why it works. When assignment is random, every hidden factor that might confound the result — how disciplined a student is, how supportive their home environment is, how much they slept the night before — gets split roughly evenly between the two groups purely by chance, because chance doesn't know or care about any of those factors. With confounders balanced between the groups, any difference you measure in the outcome afterward can be credited to the one thing that was deliberately different between the groups: the treatment itself.

Suppose your class of 20 students wants to test whether a new "active recall" revision technique (writing answers from memory, then checking) improves test scores compared to the usual re-reading of notes. Here is how you would set that experiment up in code:

import random

students = ["S1","S2","S3","S4","S5","S6","S7","S8","S9","S10",
            "S11","S12","S13","S14","S15","S16","S17","S18","S19","S20"]

random.shuffle(students)          # randomize the order

treatment_group = students[:10]   # uses active recall
control_group   = students[10:]   # keeps re-reading notes

# ... both groups sit the same test one week later ...

def average(scores):
    return sum(scores) / len(scores)

treatment_scores = [78, 82, 91, 65, 88, 74, 95, 80, 69, 84]
control_scores   = [70, 66, 75, 60, 72, 68, 79, 71, 64, 73]

print(average(treatment_scores))  # 80.6
print(average(control_scores))    # 69.8

Because random.shuffle assigned students to groups without regard to how hardworking, well-rested, or naturally strong at the subject they were, the roughly 11-point gap between the two averages (80.6 vs 69.8) is much more trustworthy evidence that active recall causes better scores than the AC/ice-cream correlation was evidence that ACs cause ice-cream demand. This is the crucial difference between this chapter's two datasets: the AC/ice-cream numbers were observed as they naturally occurred, while the revision-technique numbers came from a study where the researcher actively intervened and controlled who got what, specifically to rule out confounding.

Three Levels of Causal Reasoning

The computer scientist Judea Pearl — a UCLA professor whose work on causal inference and Bayesian networks won him the 2011 Turing Award, computing's highest honour — describes causal reasoning as a three-rung ladder, laid out in his book The Book of Why (2018, with Dana Mackenzie):

• Rung 1 — Association ("seeing"): What do I observe? This is where correlation lives. "AC sales and ice-cream sales rise together" is a Rung-1 statement.
• Rung 2 — Intervention ("doing"): What happens if I actively change something? This is where the randomized experiment lives. "If I force half the class to use active recall, their scores rise" is a Rung-2 statement — it requires action, not just observation.
• Rung 3 — Counterfactual ("imagining"): What would have happened, to this specific case, if things had gone differently? "Would this particular student have scored lower if they had re-read notes instead of using active recall?" is a Rung-3 statement — it's about a single case that can never actually be re-run, only reasoned about.

Most of the data any program collects automatically — clicks, purchases, sensor logs — sits on Rung 1. Getting to Rung 2, real causal knowledge, requires either running a controlled experiment like the one above, or (when experiments are impossible or unethical) using more advanced statistical techniques designed to approximate one from observational data — a topic that goes beyond this chapter, but that all builds on the correlation-versus-causation distinction you've just learned.

Why This Matters for Programmers, Specifically

This is not just a statistics lesson dropped into a programming chapter — it is a warning about a mistake that shows up constantly in code that analyses data. An e-commerce app's analytics might show that sales spiked the week a new banner ad ran — but if that week was also the run-up to Diwali, the festival is the confounder, and crediting the banner is exactly the AC/ice-cream error wearing a business-report disguise. A fitness app might find that users who open the app more often lose more weight, and market this as "the app causes weight loss" — but it's equally plausible that people who are already more motivated to lose weight both open the app more and lose more weight, motivation being the confounder here, not app usage causing anything. A machine learning model trained purely to predict outcomes from correlations — without any causal reasoning — will happily learn these spurious patterns and act on them, because nothing in ordinary training data tells the algorithm which correlations are causal and which are coincidental. Recognising this gap, and knowing that a randomized experiment (often called an A/B test in software companies) is the standard tool for closing it, is a core piece of engineering judgement, not an optional extra.

Check Your Understanding

Work through each scenario using the checklist from this chapter: is this confounding, reverse causation, coincidence, or genuine evidence of causation from a controlled experiment?

• A study finds that people who sleep with their shoes on tend to wake up with headaches more often. Should you conclude that sleeping with shoes on causes headaches?
• A school finds that students who sit in the front row score higher marks on average. Should the school force all students to sit in the front row?
• An app randomly shows half its users a new onboarding screen and half the old one, then finds the new-screen group has 15% higher day-7 retention. Is this good evidence the new screen caused the retention increase, and why is it different from the front-row example above?
• Using the comovement function from this chapter, write out by hand what you'd expect if you compared "number of umbrellas sold" and "number of raincoats sold" across rainy and dry months. What is the likely confounder?

Brief hints: shoes-and-headaches likely share a hidden cause — both can result from drinking too much the night before, making sleep-hygiene the confounder, not the shoes. Front-row seating is observational data, not a randomized experiment — more motivated, engaged students may simply choose to sit at the front, making motivation the likely confounder, so forcing seating changes would not automatically transfer their higher scores to everyone else. The onboarding-screen result is different because the assignment was random, ruling out confounding by design, which is exactly what makes A/B tests trustworthy evidence of causation where the front-row observation is not. Umbrellas and raincoats would show a strong positive comovement score, with rainfall as the shared confounding cause of both purchases.

Summary

Correlation is a measurable, honest fact about how two quantities move together in your data — you can compute it directly, as the comovement function did, from nothing more than sums and averages. Causation is a much stronger claim: that changing one quantity actually produces the change in the other. Every correlation you meet could be explained by real causation, reverse causation, a hidden confounding variable, or plain coincidence — and the raw numbers alone cannot tell you which. The one reliable way to isolate true causation is the randomized controlled trial: assign the treatment randomly, so that every confounder gets balanced away by chance rather than smuggled in by whoever chose who got what. As a programmer working with data — analytics dashboards, experiment logs, machine-learning training sets — the discipline of asking "correlation, or causation?" before you act on a pattern is what separates a careful engineer from one who ships a banner ad and calls it Diwali.

Think About It

Think about this: How would you explain causal inference: understanding cause and effect to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where causal inference: understanding cause and effect is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting causal inference: understanding cause and effect to at least 3 other topics you have studied.