A/B Testing: Statistical Experiments

An online bookstore in Pune is running a festival sale. Someone on the design team swaps the plain grey "Buy Now" button for an orange one carrying a "Free Shipping" badge. By the end of the week, the orange button's page converted 6.25% of visitors into buyers, while the grey button's page converted only 5%. The team declares victory and ships the orange button everywhere.

Here is the uncomfortable question this chapter is built around: how do you know that 1.25-percentage-point gap is real, and not just the ordinary noise you'd get even if the two buttons were identical? If you showed the grey button to two different random groups of 4,000 visitors on two different days, you would almost never see exactly 5.00% conversion both times either — chance alone moves the number around. A/B testing is the discipline of designing an experiment and doing the arithmetic that lets you tell a genuine effect apart from that background noise. It is one of the few places in a data science pipeline where a wrong answer costs a company real money and a right answer is provable with algebra you already know.

Why "Just Compare the Numbers" Fails

Before building the test, it helps to see why casual comparison is dangerous — this is where most beginners, and a lot of working analysts, go wrong.

Suppose instead of running both buttons at the same time, the bookstore had used the grey button in September and the orange button in October. October's conversion rate came out higher. Does that prove the orange button is better? No — October also had payday for most salaried customers, cooler weather driving more online shopping, and a marketing email campaign the team forgot about. Time itself is a confounding variable: something that changes alongside your treatment and offers a rival explanation for the result. This is the core weakness of an observational comparison — you observe two groups that differ in the one thing you care about and in a dozen things you don't.

An experiment fixes this with one deliberate move: random assignment. Every visitor arriving at the site during the test window is randomly routed to either the grey-button page (the control group, A) or the orange-button page (the treatment group, B), by a coin flip the visitor never sees. Because the assignment is random and happens over the same days, to the same traffic mix, everything else that could affect conversion — time of day, device, city, payday, marketing emails — gets spread roughly evenly across both groups by the law of averages. The only systematic difference left between group A and group B is the button. If the conversion rates still differ by more than chance alone would produce, the button is the only explanation left standing. This is why A/B testing is called a controlled experiment, and why it can support a causal claim ("the badge caused more purchases") that an observational comparison never can.

Setting Up the Test: Notation

Every visitor in each group either buys (a "success") or doesn't. This is a Bernoulli trial. If the true, unknown probability that a visitor with the grey button converts is p_A, and the true probability with the orange button is p_B, then across n_A independent visitors in group A, the number of purchases X_A follows a Binomial distribution: X_A ~ Binomial(n_A, p_A). Same for group B.

We never get to know p_A and p_B directly — they are properties of the entire population of possible visitors. What we observe is a sample estimate:

• p̂_A = x_A / n_A — the observed conversion rate in the control group
• p̂_B = x_B / n_B — the observed conversion rate in the treatment group

The whole question of the chapter can now be written precisely: is the gap p̂_B − p̂_A we observed large enough that it's implausible under the assumption that p_A = p_B — i.e., that the button genuinely makes no difference and every visitor, no matter which page they saw, had the same underlying chance of buying?

Why Chance Alone Creates a Gap

This is the piece Khan-Academy-style treatments usually skip, and it's the piece that makes the rest of the chapter make sense. We need to know how much p̂_A wobbles from sample to sample, purely from randomness, even when p_A is fixed.

For a Binomial random variable X ~ Binomial(n, p), two standard results (which you can derive by summing over the n independent Bernoulli trials that make up X, since variance adds across independent trials) are:

E[X] = np     Var(X) = np(1 − p)

Our estimate is p̂ = X/n, a rescaling of X. Rescaling a random variable by a constant c multiplies its variance by c², so with c = 1/n:

Var(p̂) = Var(X/n) = (1/n²)·Var(X) = (1/n²)·np(1 − p) = p(1 − p)/n

Taking the square root gives the standard error of a sample proportion:

SE(p̂) = √( p(1 − p) / n )

Notice the n sits inside a square root. To cut the wobble in half, you don't need double the visitors — you need four times as many. This single fact is why A/B tests on low-traffic pages take so long to produce a trustworthy answer, and it is the reason the sample-size discussion later in this chapter matters in practice, not just in theory.

Now, since group A and group B are sampled independently (a visitor in one group has no effect on a visitor in the other), the variance of their difference is the sum of their individual variances — a general rule for independent random variables: Var(X − Y) = Var(X) + Var(Y) whenever X and Y are independent. Applying it to p̂_B − p̂_A:

Var(p̂_B − p̂_A) = p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B

So even if the button changes nothing at all — even if p_A = p_B exactly — the observed gap p̂_B − p̂_A is not expected to be exactly zero. It is expected to scatter around zero with a spread given by the square root of the expression above. The entire job of an A/B test is to measure that expected scatter and check whether the gap we actually saw is bigger than the scatter can plausibly explain.

The Hypotheses

We formalize "the button changes nothing" as the null hypothesis:

H₀ : p_A = p_B

and the claim the design team is hoping for as the alternative hypothesis:

H₁ : p_A ≠ p_B

This is a two-tailed test — we allow for the possibility that the new button could make things worse, not just better, and we want to catch either. (If the business question were only ever "is the new button an improvement, yes or no," you would instead use a one-tailed test, H₁ : p_B > p_A, which uses a different, less conservative cutoff. Two-tailed is the safer default whenever a negative surprise is possible, and it's what we use below.)

Building the Test Statistic

Under H₀, both groups share one true conversion probability. We don't know it, but our best combined estimate — using all 8,000 visitors from both groups — is the pooled proportion:

p̂_pool = (x_A + x_B) / (n_A + n_B)

We use this pooled value (instead of p̂_A and p̂_B separately) inside the variance formula, precisely because H₀ assumes there's only one true rate to estimate. This gives the pooled standard error:

SE_pool = √( p̂_pool(1 − p̂_pool) · (1/n_A + 1/n_B) )

Finally, we standardize the observed gap by dividing it by that expected scatter, producing a z-statistic:

z = (p̂_B − p̂_A) / SE_pool

Why is this useful? Because when n_A and n_B are both reasonably large (a rule of thumb used in CBSE and JEE probability problems: np ≥ 10 and n(1 − p) ≥ 10), a Binomial distribution is very well approximated by a Normal distribution — a fact known since 1733 as the De Moivre–Laplace theorem, the earliest special case of what we now call the Central Limit Theorem. That means, under H₀, our z behaves like a draw from the standard normal distribution N(0,1): centered at 0, with about 95% of its values falling between −1.96 and +1.96. So we have a ruler. If the z we compute from real data lands far outside that band, either something genuinely rare happened by chance, or — far more plausibly — H₀ is false and the button really does change behavior.

Worked Example: The Free-Shipping Badge

Back to the bookstore. Over the sale week, 4,000 visitors were randomly shown each page.

• Grey button (A): n_A = 4000, x_A = 200 purchases, so p̂_A = 200/4000 = 0.05 (5%)
• Orange badge (B): n_B = 4000, x_B = 250 purchases, so p̂_B = 250/4000 = 0.0625 (6.25%)

Step 1 — pooled proportion:

p̂_pool = (200 + 250) / (4000 + 4000) = 450/8000 = 0.05625

Step 2 — pooled standard error:

SE_pool = √( 0.05625 × 0.94375 × (1/4000 + 1/4000) ) = √( 0.0530859 × 0.0005 ) = √0.0000265430 ≈ 0.005152

Step 3 — the z-statistic:

z = (0.0625 − 0.05) / 0.005152 = 0.0125 / 0.005152 ≈ 2.43

Here is the same calculation as code, so you can trace it and confirm it yourself:

def z_test_two_proportions(x_A, n_A, x_B, n_B):
    p_A = x_A / n_A
    p_B = x_B / n_B
    p_pool = (x_A + x_B) / (n_A + n_B)
    se = (p_pool * (1 - p_pool) * (1/n_A + 1/n_B)) ** 0.5
    z = (p_B - p_A) / se
    return round(z, 2)

print(z_test_two_proportions(200, 4000, 250, 4000))
# 2.43

Since |2.43| > 1.96, the observed gap is bigger than random noise alone would typically produce if H₀ were true. We reject H₀ at the conventional 5% significance level: the orange badge appears to genuinely change conversion behavior, not just appear to by luck.

We can go one step further and build a 95% confidence interval for the true difference p_B − p_A, using the unpooled standard error (appropriate once we're estimating the effect size rather than testing against H₀):

SE_unpooled = √( 0.05(0.95)/4000 + 0.0625(0.9375)/4000 ) ≈ 0.00515

0.0125 ± 1.96 × 0.00515  ⇒  [0.0024, 0.0226]

Translated: we are 95% confident the badge genuinely lifts conversion by somewhere between 0.24 and 2.26 percentage points. That interval is entirely above zero, which is the confidence-interval way of confirming the same conclusion as the z-test — but it also tells the team something the z-test alone doesn't: the effect might be as small as a quarter of a percentage point, information they need before deciding whether the badge is worth the engineering effort to ship permanently.

The Null Distribution, Visually

The diagram below shows the standard normal curve — the distribution z would follow if the button truly made no difference. The shaded tails, beyond −1.96 and +1.96, together hold exactly 5% of the area. Our observed z ≈ 2.43 (dashed green line) lands inside the right-hand shaded tail — which is exactly why we reject H₀: a value that extreme is too rare under "no real effect" to be a comfortable explanation.

What a p-value Actually Means

The p-value is the probability, computed assuming H₀ is true, of seeing a result at least as extreme as the one observed. For our two-tailed test with z ≈ 2.43, the p-value works out to about 0.015 — roughly a 1.5% chance of a gap this large or larger showing up purely from random assignment noise, if the badge truly changed nothing.

Two things this is not, and this is the single most common misreading of statistics in every field that uses it: the p-value is not "the probability that H₀ is true," and it is not "the probability the result is due to chance." It is a statement about how surprising the data would be if H₀ were true — it says nothing directly about how likely H₀ itself is. The conventional threshold of α = 0.05 (a 5% significance level) is a human convention, not a law of mathematics — physicists searching for new particles, where a false discovery claim is extremely costly to a field's credibility, require a "five-sigma" result (roughly a 1-in-3.5-million false-positive rate) before calling something a discovery, exactly the same logic as here with a far stricter bar.

Two Ways to Be Wrong

Rejecting or failing to reject H₀ is a decision made under uncertainty, so two distinct mistakes are possible:

• Type I error (false positive): H₀ is actually true (the button really does nothing), but our sample happened to produce |z| > 1.96 anyway, so we wrongly declare a winner. By construction, this happens with probability exactly α — that's what "significance level" means.
• Type II error (false negative): H₀ is actually false (the button really does help), but our sample wasn't extreme enough to cross the 1.96 threshold, so we wrongly conclude "no effect found." Its probability is called β.

1 − β is called the statistical power of the test — the probability of correctly detecting a real effect of a given size. Power depends on three things you can control or reason about before running the test: the sample size n, the significance level α you've chosen, and the true effect size you're trying to detect (bigger real effects are easier to detect than small ones).

Why Sample Size Is Not a Detail — It's the Whole Ballgame

Recall that SE shrinks only as √n, and a real effect only becomes detectable once it's several standard errors wide. Combining those two facts, statisticians derived an approximate formula for the sample size needed per group to reliably detect (with power 1 − β, at significance α) a true difference between p_A and p_B:

n ≈ (z_α/2 + z_β)² × [ p_A(1 − p_A) + p_B(1 − p_B) ] / (p_B − p_A)²

Plug in numbers to see why this matters in practice. Suppose the bookstore's baseline is p_A = 0.05 and the true (unknown, hoped-for) improvement is only 0.5 percentage points, p_B = 0.055 — a smaller, more realistic effect than our badge example. Using z_0.025 = 1.96 and z_0.20 = 0.84 (the standard choice for 80% power):

n ≈ (1.96 + 0.84)² × [0.05(0.95) + 0.055(0.945)] / (0.005)² = (7.84 × 0.099475) / 0.000025 ≈ 31,200 visitors per group

Over 62,000 visitors total — to reliably catch a half-percentage-point improvement. This is precisely why real companies run A/B tests for weeks, not hours: small, realistic effects genuinely require large samples, and no amount of clever analysis substitutes for enough data. This formula, and the reasoning behind it, sits squarely in territory tested by GATE-level engineering statistics and referenced in applied-mathematics extensions of the CBSE Class 12 syllabus.

Misconception: "Statistically Significant" Means "Big and Important"

This is worth stating explicitly because it trips up even experienced analysts: statistical significance only tells you the gap is unlikely to be pure noise — it says nothing about whether the gap is large enough to matter. With enough traffic, even a truly tiny, commercially meaningless effect will eventually cross z = 1.96, because SE keeps shrinking as n grows. Imagine the bookstore tested a button with 2 million visitors per group and found p̂_A = 5.00% versus p̂_B = 5.03%. With that much data, the SE is small enough that this 0.03-percentage-point gap could easily be "statistically significant" — and yet completely irrelevant to the business, not worth the cost of writing and maintaining new code. Always report both the significance test and the confidence interval for the effect size, and then ask a separate, human question: is an effect of this size, at its plausible low end, worth acting on? Significance answers "is this real?" Practical significance answers "does this matter?" — and a rigorous analyst never lets the first stand in for the second.

The Peeking Problem

One more genuine pitfall, common enough that it deserves its own name: checking your test's p-value every single day and stopping the moment it first dips below 0.05. This inflates your true false-positive rate far above the 5% you intended, because you are effectively giving random noise many separate chances to cross the threshold by luck — and it only needs to succeed once. A test correctly designed to have a 5% false-positive rate over its planned duration can end up with a false-positive rate of 20–30% or more if you peek daily and stop early on the first "win." The fix is to decide the sample size (or the calendar duration) in advance, based on the power calculation above, and commit to analyzing the result only once that target is reached — or, if early looks are unavoidable, to use a sequential-testing correction designed for exactly this situation, a topic that extends this chapter's ideas into research-level experimental design.

Active Recall

1. A food-delivery app tests a new checkout screen: n_A = 2500, x_A = 150; n_B = 2500, x_B = 190. Compute p̂_A, p̂_B, the pooled SE, and z. Is the result significant at α = 0.05?
2. Explain, in your own words, why a random assignment mechanism is necessary for an A/B test to support a causal claim — and give an example of a confounding variable it protects against.
3. A colleague says, "Our p-value was 0.03, so there's a 97% chance our new feature actually works." What is wrong with this statement, precisely?
4. Why does halving the standard error of a proportion require quadrupling the sample size rather than doubling it? Show the algebra.
5. Define Type I and Type II error for the specific scenario of testing whether a new UPI payment button increases successful transactions. What real-world cost does each type of mistake carry for the company?
6. A test run for a single day shows z = 2.1, crossing significance. The team is tempted to stop immediately. What risk are they taking on, and what should they do instead?

Answer key (Q1): p̂_A = 0.06, p̂_B = 0.076, p̂_pool = 340/5000 = 0.068, SE_pool = √(0.068 × 0.932 × (1/2500 + 1/2500)) ≈ 0.00712, z = 0.016/0.00712 ≈ 2.25. Since 2.25 > 1.96, the result is significant at the 5% level.

Summary

An A/B test is a controlled experiment: randomly split traffic into a control group and a treatment group so that the only systematic difference between them is the one variable you changed, then measure a conversion proportion in each. Because both proportions are estimates built from random samples, they scatter around their true values with a standard error of √(p(1−p)/n) — a quantity that shrinks only as the square root of the sample size. To decide whether an observed gap between groups is a real effect or just this expected scatter, standardize it into a z-statistic using the pooled standard error, and compare it against ±1.96 for a two-tailed test at the conventional 5% significance level. A p-value below 0.05 lets you reject the null hypothesis that the groups are identical — but it is a statement about the data's surprise under "no effect," never a probability that the effect is real, and never, by itself, a statement about whether the effect is large enough to matter. Guard against two further pitfalls that no formula fixes automatically: running the test with too small a sample to have any real power to detect the effect you care about, and peeking at results early and stopping the moment they look favorable. Together, randomization, the standard-error calculation, and a pre-committed sample size are what separate an A/B test from an anecdote.