Experimental Design and A/B Testing

An electronics retailer in Pune sends a 10% discount coupon by email on the first day of Diwali week. By the end of the week, bookings are up 34% compared to the week before. The marketing team declares victory: the coupon worked. But Diwali week is when Indian households buy electronics anyway — new appliances for the home, gifts, upgrades before the festive season ends. Sales might have risen 34%, 20%, or even 40% with no coupon at all. The team has no way to know, because they never created a version of the week without the coupon to compare against. This is the central problem this chapter solves: how do you set up a comparison so that when you see a difference, you can actually trust that your change caused it, and not something else that happened to change at the same time?

That "something else" has a name: a confounding variable, a factor that changes alongside the thing you're testing and offers a rival explanation for the result. Diwali timing is a confound. So is a cricket final airing the same evening as your app update, or a school raising its cutoff the same year it changed its teaching method. Whenever you compare "before" to "after," or "people who did X" to "people who didn't," without controlling what else was different, you are doing an observational comparison — and observational comparisons can suggest causes, but they cannot prove them. The tool that can is the controlled experiment, and its digital descendant is the A/B test, the method that decides which version of a website, app screen, or notification actually performs better, used by every product team at scale — from UPI apps deciding button colours to IRCTC-style booking flows deciding form layouts.

What Makes a Comparison Fair

Suppose you want to know whether a new teaching method raises Class 10 board scores. The wrong way: teach it to whichever students volunteer, and compare their marks to everyone else's. This fails because volunteers are probably already more motivated, and motivation — not the method — may explain any score gap. This is selection bias: the groups differed in important ways before the experiment even began.

The fix, worked out rigorously by the statistician Ronald A. Fisher at the Rothamsted Experimental Station in England in the 1920s while designing agricultural field trials, rests on three principles that still define experimental design a century later:

• Randomization. Assign subjects to groups using a random mechanism — a coin toss, a random number generator — not by choice, convenience, or self-selection. Randomization doesn't guarantee any two groups are identical, but it guarantees that, on average, every factor you didn't measure (motivation, prior knowledge, family income, internet speed) gets distributed roughly evenly across groups, purely by the law of large numbers. Any factor that could confound your result is now equally likely to land in either group.
• Control. Keep one group unchanged (the control group) as a baseline, and change exactly one thing for the other (the treatment or variant group). If ten things differ between your groups, you cannot tell which one caused the outcome.
• Replication. Run the comparison on enough independent subjects that a real effect isn't lost in random noise, and a fluke isn't mistaken for a real effect. We will make this precise with mathematics shortly — "enough" has an exact formula, not a guess.

Fisher's own experiments compared crop yields across differently fertilized plots of land, randomly assigning which plot got which fertilizer so that soil quality, sunlight, and drainage — factors no one could fully control — would even out across treatments. The same three principles, unchanged, now decide what colour your "Buy Now" button is.

A/B Testing: Fisher's Idea, One Button at a Time

An A/B test is a randomized controlled experiment run on a digital product. You take a stream of users, randomly split them into two (or more) groups, show group A the existing version (control) and group B a version with exactly one change (variant), and measure a chosen outcome — called the metric — for each group. If the metric differs by more than random chance would produce, you have evidence the change caused the difference.

Here is a concrete run: a payments app is deciding whether to change its checkout button from a plain blue "Pay" label to a green "Pay Now" label with slightly more urgent copy. Over one week, 10,000 users who reach the payment screen are randomly and independently split: 5,000 see the blue button (Group A, control), 5,000 see the green button (Group B, variant). Each user's assignment is like an independent coin flip — nothing about the user (device, city, past behaviour) influences which button they see.

Results: in Group A, 620 of 5,000 users complete the payment. In Group B, 690 of 5,000 complete it. So the observed conversion rates are:

p_A = 620 / 5000 = 0.1240  (12.40%)
p_B = 690 / 5000 = 0.1380  (13.80%)
observed difference = p_B - p_A = 0.0140  (1.40 percentage points)

A 1.4 percentage point gain looks encouraging — it's an 11.3% relative improvement (0.014 / 0.124). But before anyone changes the button for every user in India, we need to answer a sharper question: could a gap this size show up just from randomly splitting one identical population into two groups, even if the button makes no real difference at all? This is where the mathematics of the two-proportion test comes in.

Is the Difference Real? The Two-Proportion Z-Test, Derived

Model each user in Group A as an independent Bernoulli trial: they either pay (1) or don't (0), with true probability p_A. If X is the count of payers among n_A users, X follows a Binomial(n_A, p_A) distribution. Two standard facts about the binomial distribution, both on the CBSE Class 12 probability syllabus, do all the work:

Mean of X:      E[X] = n * p
Variance of X:  Var(X) = n * p * (1 - p)

The observed proportion is p̂ = X / n, a rescaling of X. Rescaling a random variable by a constant c multiplies its variance by c², so:

Var(p̂) = Var(X / n) = Var(X) / n^2 = [n * p * (1-p)] / n^2 = p(1-p) / n

This is the standard error of a single proportion, squared. Now, Group A and Group B are independent — no user appears in both, and the random assignment means what happens in one group tells you nothing about the other. For independent random variables, variances of a sum (or difference) add:

Var(p̂_B - p̂_A) = Var(p̂_B) + Var(p̂_A) = p_B(1-p_B)/n_B + p_A(1-p_A)/n_A

SE = sqrt[ p_A(1-p_A)/n_A + p_B(1-p_B)/n_B ]

By the Central Limit Theorem, for n this large (thousands of trials, with n·p and n·(1−p) both comfortably above the usual rule-of-thumb of 5–10), the sampling distribution of p̂_B − p̂_A is closely approximated by a Normal distribution. So we standardize the observed difference into a z-score:

z = (p̂_B - p̂_A) / SE

If the button genuinely makes no difference (the null hypothesis, H₀: p_A = p_B), z behaves like a draw from a standard Normal(0, 1) distribution. Under a standard Normal curve, only 5% of the area lies beyond ±1.96 — so if our computed |z| exceeds 1.96, the observed gap would be a rare (less-than-1-in-20) coincidence under H₀, and we call the result statistically significant at the 95% confidence level and reject H₀ in favour of a real effect.

Let's compute it for our button test, in code that traces exactly what the algebra above says:

import math

def two_proportion_z(x_a, n_a, x_b, n_b):
    p_a = x_a / n_a
    p_b = x_b / n_b
    se = math.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
    z = (p_b - p_a) / se
    return p_a, p_b, se, z

p_a, p_b, se, z = two_proportion_z(620, 5000, 690, 5000)
print(f"p_A = {p_a:.4f}, p_B = {p_b:.4f}")
print(f"SE  = {se:.5f}")
print(f"z   = {z:.3f}")

# Output:
# p_A = 0.1240, p_B = 0.1380
# SE  = 0.00675
# z   = 2.075

Trace it by hand to confirm: p_A(1−p_A)/n_A = 0.124 × 0.876 / 5000 = 0.0000217; p_B(1−p_B)/n_B = 0.138 × 0.862 / 5000 = 0.0000238. Their sum is 0.0000455, and its square root is 0.00675. Dividing the observed gap, 0.0140, by 0.00675 gives z ≈ 2.075. Since 2.075 exceeds the 1.96 threshold, this result is statistically significant at 95% confidence — the two-tailed p-value works out to roughly 0.04, meaning a gap this large (or larger) would occur by pure chance only about 4 times in 100 identical experiments if the button truly made no difference. The product team has real evidence, not a Diwali-week illusion.

How Many Users Do You Need? Designing Before You Run

The z-test above is an after-the-fact analysis — it tells you whether data you already collected shows a real effect. Good experimental design asks the question in advance: if the true lift is around some size δ, how many users per group do I need to have a good chance of detecting it, before I even start collecting data?

Under H₀, z is approximately Normal(0, 1), and we reject when |z| exceeds the critical value z_α/2 (1.96 for 95% confidence). If a real effect of size δ = p_B − p_A exists, z is approximately Normal(δ/SE, 1) — shifted right by δ/SE. We want to choose n large enough that this shifted distribution puts most of its mass past the threshold; specifically, enough that we detect the effect with probability (1−β), the test's power (commonly 80%, so β = 0.20). This requires:

delta / SE = z_(alpha/2) + z_beta

Using a pooled baseline rate p̄ (our best guess of the typical conversion rate before the test) and equal group sizes n, SE ≈ √(2p̄(1−p̄)/n). Substituting and solving for n:

sqrt(2 * p_bar * (1-p_bar) / n) = delta / (z_(alpha/2) + z_beta)

n = 2 * p_bar * (1 - p_bar) * (z_(alpha/2) + z_beta)^2 / delta^2

For 95% confidence (z_α/2 = 1.96) and 80% power (z_β ≈ 0.84), suppose the team's pooled baseline before running the button test was p̄ = (620+690)/10000 = 0.131, and they hoped to detect a lift of δ = 0.014, exactly what they eventually observed:

import math

def required_sample_size(p_bar, delta, z_alpha=1.96, z_beta=0.84):
    numerator = 2 * p_bar * (1 - p_bar) * (z_alpha + z_beta) ** 2
    return math.ceil(numerator / delta ** 2)

n_needed = required_sample_size(p_bar=0.131, delta=0.014)
print(n_needed)

# Output: 9108

A proper 80%-power design for this effect size needs about 9,108 users per arm — nearly double the 5,000 actually used. That is not a contradiction of the earlier significant result; it is an important, honest distinction. A power calculation says "with this many users, I will detect a true effect of this size 80% of the time" — it does not say a smaller sample can never detect it. With only 5,000 per arm, this test had somewhere near 50–55% power for a true 1.4-point lift, meaning it was roughly a coin flip whether it would come back significant at all. It happened to land on the "detected" side. A well-run product team would treat this result as encouraging but would want to replicate it, or plan future tests using the 9,108-per-arm figure, rather than treat one lucky significant result as the final word.

Three Ways A/B Tests Go Wrong

Misconception 1: "A bigger percentage-point gap is always more believable." Believability depends on sample size, not just gap size. Compare two hypothetical tests with an almost identical relative pattern: Test X, n = 100 per group, p_A = 0.12 (12/100), p_B = 0.14 (14/100), a 2-point gap. Here SE = √(0.12×0.88/100 + 0.14×0.86/100) = √(0.001056 + 0.001204) = √0.00226 ≈ 0.0475, giving z = 0.02/0.0475 ≈ 0.42 — nowhere near significant; this gap is entirely consistent with random noise. Yet our earlier button test had a smaller gap (1.4 points, not 2) and was significant, purely because it used 50 times more users. Sample size, not the raw size of the gap, is what makes a comparison trustworthy.

Misconception 2: "I'll just check the dashboard each morning and stop the moment p drops below 0.05." This is the optional stopping or repeated significance testing problem, and it is one of the most common real errors in industry A/B testing. Each day you check, you are effectively running a new test on partially overlapping data. Even if the button truly has zero effect, pure randomness will make z cross 1.96 at some point during a multi-week test far more often than 5% of the time — because you gave randomness 14 or 20 separate chances to get lucky, instead of one. This is a special case of the general multiple comparisons problem: testing many times and reporting only the significant hit inflates your true false-positive rate well above the nominal 5%. The correct practice is to fix the sample size (using the power calculation above) and the analysis date in advance, and look only once.

Misconception 3: "Comparing users who opted into a new feature against users who didn't is basically an A/B test." It is not — it is an observational comparison wearing an A/B test's clothes. Users who voluntarily opt in typically differ systematically from those who don't (more engaged, more tech-comfortable, checking the app more often anyway), so any gap you find may reflect who chose the feature, not the feature itself. This is exactly the selection-bias failure from the teaching-method example earlier. A true A/B test requires the platform, not the user, to make the random assignment.

A fourth, subtler point worth naming: statistical significance is not the same as practical significance. Because SE shrinks proportionally to 1/√n, sufficiently large samples can make even a trivial, commercially meaningless difference (say, 0.05 percentage points) statistically significant. A rigorous experimenter always asks two separate questions: is this difference real (statistical significance), and is this difference large enough to matter (practical significance, often called the effect size)? Our button test passed both bars — 11.3% relative lift is both statistically detectable and commercially worth shipping.

Diagram: How One A/B Test Flows

Beyond One Button: Blocking, and Where This Sits in Your Syllabus

Real experiments often can't fully randomize away every confound — sometimes you know in advance that a variable matters (say, soil type across fields, or device type across app users) and want to control for it directly rather than hope randomization balances it out. Fisher's third classical technique, blocking (formally, Randomized Block Design), first groups subjects into "blocks" of similar units — plots with the same soil type, users on the same device class — and randomizes treatment assignment separately within each block. This is still standard practice today: agricultural variety trials run by ICAR institutes and by the Indian Agricultural Statistics Research Institute (IASRI) in New Delhi use randomized block designs to compare new crop varieties or fertilizer treatments across test plots, exactly following Fisher's Rothamsted logic a century later. The same principle survives in tech A/B testing when a team "stratifies" its random split by device type or city before assigning users, to guarantee both arms have matching device mixes rather than trusting randomization alone with a smaller sample.

The mathematics in this chapter maps directly onto material you will meet again, more formally, at higher levels. CBSE Class 12 Mathematics builds the Bernoulli and binomial distribution machinery (mean np, variance np(1−p)) used to derive the standard error above; JEE Main and Advanced regularly test binomial mean/variance and probability calculations of exactly this flavour; BITSAT's quantitative and data-interpretation sections reward the same comfort with proportions and rates; and Olympiad-style or KVPY-style problems often hide a confounding-variable fallacy inside a word problem, testing whether you notice the missing control group rather than whether you can compute an average. At the GATE level, "Design of Experiments" and hypothesis testing appear explicitly in the Probability and Statistics portion of the Data Science & AI syllabus, extending precisely the z-test and power-analysis reasoning built here to more general tests (t-tests for small samples, ANOVA for more than two groups). Even outside computing, India's Clinical Trials Registry, maintained under the Indian Council of Medical Research, requires new drug trials to be registered as randomized controlled trials before they can be conducted — the same three Fisherian principles, applied to human health rather than button colours.

Summary

• An observational comparison (before/after, opt-in/opt-out) can be misled by confounding variables that changed alongside your treatment; only a controlled experiment with random assignment lets you claim causation.
• Fisher's three principles — randomization, control, replication — still define a valid experiment, whether the units are farm plots or app users.
• An A/B test randomly splits users into a control (A) and variant (B) group differing in exactly one change, then compares a metric between them.
• For proportions, SE = √[p_A(1−p_A)/n_A + p_B(1−p_B)/n_B], derived directly from the variance of a rescaled binomial count; z = (p̂_B − p̂_A)/SE; |z| > 1.96 signals significance at 95% confidence.
• Required sample size for a target effect δ and power grows as n = 2p̄(1−p̄)(z_α/2+z_β)²/δ² — bigger for smaller effects, smaller baseline rates near 50%, and higher desired power.
• Common failure modes: judging significance by eye instead of computing z; repeatedly peeking at results and stopping early (inflates false positives); comparing self-selected groups instead of randomly assigned ones; and confusing statistical significance with practical importance.

Practice

1. A college's test-registration page is redesigned. Old form (A): n = 2000, 180 completions. New form (B): n = 2000, 210 completions. Compute p_A, p_B, SE, and z by hand. Is the 16.7% relative improvement statistically significant at 95% confidence? (Answer key: p_A = 0.090, p_B = 0.105, SE ≈ 0.00938, z ≈ 1.60 — not significant; more data is needed before concluding the new form genuinely helps.)
2. A school claims its new app improved scores because students who chose to download it scored 8 marks higher on average than those who didn't. Name the specific flaw in this argument, and describe precisely how you would redesign it as a true experiment.
3. A startup checks its A/B test dashboard every morning for three weeks and plans to stop the test the very first day the p-value drops below 0.05. Using the idea of running the same significance test many times on the same underlying data, explain why this practice makes the true false-positive rate higher than the intended 5%.
4. Baseline conversion on a page is 20%. A team wants 95% confidence and 80% power to detect a 2-percentage-point lift. Using n = 2p̄(1−p̄)(z_α/2+z_β)²/δ² with p̄ = 0.21, estimate the required sample size per arm. (Answer key: n ≈ 6,504 per arm.)

Think About It

Think about this: How would you explain experimental design and a/b testing to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.