Hypothesis Testing and Confidence Intervals: Making Decisions with Data

A snack company prints "Net Weight: 50 g" on every packet. You buy 36 packets, weigh each one on a kitchen scale, and average them out: 48.7 g. The company is selling you less than it promised — or is it? Every real weighing process has a bit of natural variation: a few grains of namkeen more or less, a tiny difference in how the sealing machine cuts the packet. If you bought another 36 packets tomorrow, the average would come out slightly different again — maybe 49.4 g, maybe 50.2 g. So the real question isn't "is 48.7 exactly equal to 50?" — it almost never will be, even if the machine is perfectly calibrated. The real question is: is 48.7 g a difference we'd expect from ordinary random variation around a true average of 50 g, or is it large enough that something is actually wrong?

This is the exact question that engineers at quality-control labs, drug regulators approving a new medicine, ISRO scientists validating a sensor, and data scientists A/B-testing a UPI app redesign all ask, every single day. The branch of statistics that answers it rigorously — with a precise number attached to "how sure are we?" — is called hypothesis testing. Its close cousin, confidence intervals, answers a slightly different but related question: instead of just checking one claimed number, it gives you a whole range of plausible true values. By the end of this chapter you will be able to derive both from first principles, not just plug numbers into a formula.

Two Competing Stories About the World

Every hypothesis test starts with two competing explanations for what you observed, and the entire machinery of the test exists to decide, using probability, which explanation the data supports.

• The null hypothesis (H₀): the boring, default explanation — "nothing unusual is going on." In the snack example: the packing machine is correctly calibrated, and the true average weight really is 50 g; the 48.7 g we measured is just ordinary sample-to-sample variation.
• The alternative hypothesis (H₁ or Hₐ): the claim you're actually investigating — "something real is going on." Here: the true average weight is not 50 g (the machine is under-filling packets).

Notice the asymmetry: H₀ is treated like a defendant in court — presumed correct until the data provides strong enough evidence against it. We never "prove H₀ true"; we only ever ask whether the data gives us enough reason to reject it. This mirrors "innocent until proven guilty" exactly, and we'll return to that analogy shortly because it also explains the two ways a test can go wrong.

Why Sample Means Vary: The Sampling Distribution

Before we can decide whether 48.7 g is "surprising," we need to know how much a sample average is expected to bounce around, just by chance, even when H₀ is true. This requires one piece of machinery: the sampling distribution of the mean.

Suppose the population of all packets really does have true mean μ and standard deviation σ (σ measures how much individual packet weights vary — some packets naturally have a bit more or less filling). If we draw a random sample of n packets, X₁, X₂, …, Xₙ, and average them to get the sample mean x̅ = (1/n)ΣXᵢ, then x̅ itself is a random quantity — it changes every time you draw a new sample. What is its spread?

Using the fact that for independent random variables, variances add:

Var(x̅) = Var( (1/n) * ΣXᵢ )
        = (1/n²) * Var(ΣXᵢ)
        = (1/n²) * Σ Var(Xᵢ)      [independence: variance of a sum = sum of variances]
        = (1/n²) * (n * σ²)       [each Xᵢ has variance σ²]
        = σ² / n

So the standard error (the standard deviation of the sample mean itself) is:

SE = σ / √n

This single line explains something every quality-control engineer relies on: averaging shrinks variability. A single packet's weight might swing by ±3 g from packet to packet, but the average of 36 packets swings far less, because random over-fills and under-fills partly cancel out across the sample. Specifically, the spread shrinks by a factor of √n — quadruple your sample size, and the standard error is cut in half, not to a quarter. This √n relationship is exact (it follows directly from independence), and it's the reason opinion polls of 1,000–2,000 people can estimate national opinion to within a percentage point or two, without needing to survey a billion people.

The second piece is the Central Limit Theorem (CLT): even if individual packet weights aren't perfectly normally distributed, the sampling distribution of x̅ becomes approximately normal (bell-shaped) as n grows — in practice, n ≥ 30 is the commonly used rule of thumb. (If the underlying population is itself normally distributed, x̅ is exactly normal for any n.) Combining both facts:

x̅ ~ approximately Normal( μ, σ²/n )

Formalizing the Test

We now have everything needed to state a hypothesis test precisely. For the snack example:

• H₀: μ = 50 (the claimed mean is correct)
• H₁: μ ≠ 50 (the true mean differs from the claim — this makes it a two-tailed test, since we'd be suspicious whether the machine over-fills or under-fills)

If instead the concern were only "is the company shortchanging us" (not "is it possibly over-filling, which nobody would complain about"), we'd write H₁: μ < 50 — a one-tailed test, checking only one direction. Choosing one-tailed vs two-tailed must be decided before looking at the data, based on what question you're actually asking — not chosen afterward to make a result look more significant.

We also fix a significance level, α (commonly 0.05, i.e. 5%) before collecting data. α is the risk we're willing to accept of rejecting a true H₀ purely by bad luck. It's a threshold we choose, not something the data tells us.

The Test Statistic: Converting a Difference into a Standard Score

We standardize the observed sample mean the same way you'd standardize any normal variable — subtract the mean, divide by the standard deviation — but using the sampling distribution's mean (μ from H₀) and standard error:

z = (x̅ − μ₀) / (σ / √n)

This z-score tells us: how many standard errors away from the claimed mean does our sample sit? If H₀ is true, z should behave like a draw from the standard normal distribution N(0, 1) — mostly staying within about 2 standard errors of 0, only rarely straying further.

Worked example. Historical quality-control records tell us the machine's packet-to-packet standard deviation is σ = 3 g (this is a known, well-established figure from years of production data, so we can use it directly rather than estimating it from our small sample). Our sample: n = 36, x̅ = 48.7 g, testing H₀: μ = 50 against H₁: μ ≠ 50 at α = 0.05.

SE = σ/√n = 3/√36 = 3/6 = 0.5
z  = (x̅ − μ₀)/SE = (48.7 − 50)/0.5 = (−1.3)/0.5 = −2.6

For a two-tailed test at α = 0.05, the critical values that cut off the outer 2.5% in each tail of the standard normal curve are ±1.96 (a value worth memorizing — it recurs throughout statistics). Since |−2.6| = 2.6 > 1.96, our observed z falls in the rejection region: it's further from 0 than we'd expect 95% of the time under H₀. We reject H₀ — the evidence is inconsistent with the machine being correctly calibrated at 50 g.

p-values: Putting a Number on "How Surprising"

The critical-value comparison above answers reject/don't-reject, but statisticians usually report a more informative number: the p-value.

The p-value is the probability, assuming H₀ is true, of obtaining a test statistic at least as extreme as the one actually observed.

For our two-tailed test with z = −2.6:

p = 2 × P(Z > 2.6)   [double it — extreme in either direction counts]
  = 2 × 0.0047
  = 0.0093

Here's the same calculation done in Python, tracing exactly what each line computes:

import math
from scipy.stats import norm

xbar, mu0, sigma, n = 48.7, 50, 3, 36
se = sigma / math.sqrt(n)          # se = 3 / 6.0 = 0.5
z = (xbar - mu0) / se              # z = -1.3 / 0.5 = -2.6
p_value = 2 * (1 - norm.cdf(abs(z)))   # 2 * (1 - 0.99534) = 0.00932

print(f"SE = {se}")            # SE = 0.5
print(f"z = {z}")              # z = -2.6
print(f"p-value = {p_value:.4f}")   # p-value = 0.0093

Since p = 0.0093 < α = 0.05, we reject H₀ — the same conclusion as before, but now we can say precisely: if the machine really were correctly calibrated at 50 g, a sample average this far from 50 would happen only about 9 times in 1,000 batches of this size, purely by chance. That's rare enough to conclude something is actually off.

The diagram below shows exactly what's happening geometrically: the bell curve is the sampling distribution of z under H₀. The shaded tails beyond ±1.96 are the rejection region (total area 0.05). Our observed z = −2.6 lands inside the left shaded tail — visibly further out than the boundary.

A Misconception Worth Killing Now: What a p-value Is NOT

The single most common error, made even by working researchers, is reading "p = 0.0093" as "there's only a 0.93% chance H₀ is true." That is wrong, and the reasoning matters. The p-value is computed by assuming H₀ is true and then asking how likely our data would be under that assumption:

p-value = P(data this extreme  |  H₀ is true)

It is emphatically not P(H₀ is true | data this extreme). These are different conditional probabilities, and confusing them is a logical error, not just sloppy phrasing — it's the same mistake as confusing "the probability it rains, given it's cloudy" with "the probability it's cloudy, given it rains." A p-value never tells you the probability that a hypothesis is true; it only tells you how surprising your data would be if that hypothesis were true. The correct interpretation of p = 0.0093 is: "if the machine really were calibrated correctly, we'd see a sample mean this far off only 0.93% of the time" — a statement about the data, conditioned on H₀, not a statement about H₀ itself.

Two Ways a Test Can Be Wrong

No test is certain. Since we're making a yes/no decision from randomly varying data, two distinct kinds of mistakes are possible, and they trade off against each other. The courtroom analogy makes this exact: treat H₀ as "the defendant is innocent."

In the snack-packet setting: a Type I error means the machine is actually fine, but our sample randomly came out low and we wrongly halt production and recalibrate a perfectly good machine — wasted cost for the company. A Type II error means the machine really is under-filling, but our particular sample happened to look normal, so we let underweight packets keep shipping — a cost to consumers and, if a regulator catches it later, to the company's reputation. Lowering α (say from 0.05 to 0.01) makes Type I errors rarer but, for a fixed sample size, makes Type II errors more likely — you become more conservative about crying wolf, which necessarily means you're slower to catch a real problem. The only way to reduce both simultaneously is to collect more data (increase n), which shrinks the standard error and sharpens the whole test.

Confidence Intervals: The Same Idea, Turned Around

A hypothesis test asks "is this one specific claimed value (μ₀ = 50) plausible?" A confidence interval asks the more general question: "given my sample, what whole range of values for μ would I not have rejected?" We derive it directly from the same standardization.

We know that for a standard normal variable, P(−1.96 ≤ Z ≤ 1.96) = 0.95. Substitute Z = (x̅ − μ)/(σ/√n) and solve the inequality for μ:

−1.96 ≤ (x̅ − μ)/(σ/√n) ≤ 1.96
−1.96·(σ/√n) ≤ x̅ − μ ≤ 1.96·(σ/√n)          [multiply through by σ/√n]
−x̅ − 1.96·(σ/√n) ≤ −μ ≤ −x̅ + 1.96·(σ/√n)    [subtract x̅]
 x̅ − 1.96·(σ/√n) ≤  μ ≤  x̅ + 1.96·(σ/√n)     [multiply by −1, flip inequalities]

Giving the general formula for a 95% confidence interval:

CI = x̅ ± 1.96 × (σ/√n)

The multiplier changes with the confidence level you want — 90% uses 1.645, 95% uses 1.96, 99% uses 2.576 — these are just the z-values that cut off the corresponding tail area on the standard normal curve.

Worked example (continuing the snack packets):

CI = 48.7 ± 1.96 × 0.5
   = 48.7 ± 0.98
   = (47.72 g, 49.68 g)

We are 95% confident the true average packet weight lies between 47.72 g and 49.68 g. Notice that 50 g — the company's claim — does not lie inside this interval, which is exactly consistent with having rejected H₀: μ = 50 above. This is not a coincidence.

The Equivalence: CI and Two-Tailed Test Are the Same Test, Viewed Differently

This is worth stating as a theorem, because it lets you shortcut a lot of exam problems: a two-tailed hypothesis test of H₀: μ = μ₀ at significance level α rejects H₀ if and only if μ₀ falls outside the corresponding (1 − α)×100% confidence interval. Both are built from exactly the same inequality — we just solved it for a different unknown (z in one case, μ in the other). Once you've computed one, you get the other's conclusion for free.

A Second Misconception: What "95% Confidence" Actually Means

It's tempting to say "there's a 95% probability that μ lies in (47.72, 49.68)." This is subtly wrong for the same reason the p-value misconception is wrong: once the sample is collected and the interval (47.72, 49.68) is a fixed pair of numbers, μ either lies in it or it doesn't — there's no more randomness left to assign a probability to. The correct interpretation is about the procedure, not this one interval: if you repeated this sampling process — draw 36 packets, compute x̅, build the interval — many times, about 95% of the resulting intervals would contain the true μ. The randomness is in which sample you happened to draw, not in μ itself (μ is a fixed, if unknown, number). "95% confidence" is a statement about the long-run reliability of the method, not a probability statement about this specific interval.

A Third Misconception: "Fail to Reject" ≠ "Proved True"

When a test doesn't reach significance, it is tempting to say "we accept H₀" or "we proved the machine is fine." Never say this. Failing to reject H₀ only means the data didn't provide strong enough evidence against it — possibly because H₀ really is true, but possibly because your sample was too small to detect a real but modest deviation (a Type II error risk). The correct, careful phrase is always "fail to reject H₀," never "accept H₀."

Worked Example: A One-Tailed Test

Suppose a digital-payments company claims its UPI transactions complete in under 3 seconds on average. A regulator samples n = 64 transaction logs, finds x̅ = 3.15 s, and uses the platform's historical standard deviation σ = 0.8 s. This is a one-tailed claim — the company isn't claiming "exactly 3 s," it's claiming "less than 3 s" — so:

• H₀: μ ≥ 3 (no improvement; the skeptical default)
• H₁: μ < 3 (the company's claim)

SE = 0.8/√64 = 0.8/8 = 0.1
z  = (3.15 − 3)/0.1 = 0.15/0.1 = 1.5

Because H₁ only points left (μ < 3), the rejection region is only the left tail: z ≤ −1.645 at α = 0.05. Our observed z = +1.5 is nowhere near that region — in fact it's on the opposite side of 0 entirely, since the sample mean came out slightly above 3 seconds, not below. We fail to reject H₀: this sample gives no evidence that transactions average under 3 seconds. This example matters pedagogically because it shows a one-tailed test only ever rejects in its stated direction — even a large-looking gap in the "wrong" direction leads straight to "fail to reject," never to accidentally supporting the opposite claim.

CBSE and Competitive-Exam Connections

Formal hypothesis testing sits in CBSE's Applied Mathematics stream (Classes 11–12), where the "Inferential Statistics" unit is built almost entirely on the ideas in this chapter — z-tests, t-tests (used when σ is unknown and the sample is small, a natural next chapter after this one), and confidence intervals. Core Mathematics (041) doesn't formally test hypotheses, but everything here — standard deviation, the normal distribution, standardized z-scores — feeds directly into the probability distribution questions that do appear on JEE Main and Advanced, since those exams lean heavily on exactly this kind of "how much does an average vary" reasoning, even without naming it "hypothesis testing." At the GATE level, Engineering Mathematics papers across several branches include a "Probability and Statistics" component where testing of hypotheses and confidence intervals appear directly, so the derivations in this chapter — not just the formulas — are worth keeping.

Practice: Test Your Understanding

1. A tea-packaging plant claims each pouch weighs 100 g with a known σ = 4 g. A sample of 25 pouches gives x̅ = 98.2 g. Compute z for H₀: μ = 100 vs H₁: μ ≠ 100, and decide at α = 0.05 whether to reject H₀.
2. Using the same sample, construct a 95% confidence interval for the true mean pouch weight, and check that your conclusion in Q1 is consistent with whether 100 g falls inside it.
3. Explain, in your own words, why "p = 0.02" does not mean "there's a 2% chance the null hypothesis is true."
4. A hospital wants to test whether a new drug reduces average recovery time below the current standard of 10 days. Write H₀ and H₁, and state whether this should be a one-tailed or two-tailed test.
5. If you increase your sample size from n = 36 to n = 144 (four times as many), by what factor does the standard error shrink? Use the SE formula to justify your answer.
6. A test fails to reject H₀ at α = 0.05. A classmate says, "This proves the machine is perfectly calibrated." What is wrong with that statement, and what would be the correct way to phrase the conclusion?

Answer key: (1) z = (98.2−100)/(4/√25) = −1.8/0.8 = −2.25; since |−2.25| > 1.96, reject H₀. (2) CI = 98.2 ± 1.96(0.8) = 98.2 ± 1.568 = (96.63, 99.77); 100 g lies outside it, consistent with rejecting H₀. (3) A p-value is computed by assuming H₀ true and asking how likely the data would be — it says nothing about the probability that H₀ itself is true; that would require conditioning the other way around (Bayes' theorem territory, with a prior on H₀, which a plain p-value never uses). (4) H₀: μ ≥ 10, H₁: μ < 10 — one-tailed, since the claim is specifically "less than." (5) SE shrinks by a factor of √4 = 2 (from σ/6 to σ/12), not by a factor of 4 — standard error scales with 1/√n, not 1/n. (6) Failing to reject only means the sample didn't show strong enough evidence against H₀; it never proves H₀ true. The correct phrasing is "we fail to reject H₀ at α = 0.05 — the data are consistent with correct calibration, but this does not prove it."

Summary

• Sample means vary from sample to sample; their spread is the standard error, SE = σ/√n, derived exactly from Var(x̅) = σ²/n using independence.
• A hypothesis test sets up H₀ (default, presumed true) against H₁ (the claim under investigation), computes z = (x̅ − μ₀)/SE, and compares it to a critical value (±1.96 for a two-tailed test at α = 0.05).
• The p-value is P(data this extreme | H₀ true) — never the probability that H₀ is true. Reject H₀ when p < α.
• Type I error (α) = falsely rejecting a true H₀; Type II error (β) = falsely failing to reject a false H₀. They trade off against each other; only more data reduces both.
• A confidence interval, x̅ ± z·(σ/√n), gives the range of μ values a two-tailed test would not have rejected — it is the same inequality solved for a different unknown, and its "95% confidence" describes the long-run reliability of the method, not the probability that this one interval contains μ.
• Never say "accept H₀" or "proved H₀ true" — only "reject" or "fail to reject," because absence of evidence against H₀ is not evidence that H₀ is correct.

Think About It

Think about this: How would you explain hypothesis testing and confidence intervals: making decisions with data to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.