Bayesian Inference: Learning from Data

A test comes back positive. How worried should you be?

A city lab launches a rapid screening test for a disease. It is advertised as "95% accurate" — if you have the disease, it catches it 95% of the time. You take the test. It comes back positive. Most people's gut reaction is: I'm 95% likely to have this disease. That reaction is wrong, and it isn't wrong by a little — for a rare disease, the true answer can be closer to 15% than to 95%. By the end of this chapter you will be able to compute the correct number yourself, explain precisely why the gut reaction fails, and — more importantly — understand the general machinery for updating a belief whenever new evidence arrives. That machinery is called Bayesian inference, and it is one of the few ideas in mathematics that changes how you reason about uncertainty for the rest of your life, not just how you solve exam problems.

Two ways to think about probability

Up to now, probability has probably meant "long-run frequency": toss a fair coin a million times, and heads shows up close to 500,000 times. That view works fine for coins and dice, but it says nothing useful about a one-off statement like "there is a 70% chance my exam result will improve this year" or "this specific patient has a 16% chance of having the disease." Those aren't repeatable experiments — they are single events about which we hold a degree of belief.

The Bayesian view treats probability as exactly that: a measure of belief, held by a specific observer, given the evidence available so far. The power of the framework is that it gives you an exact, non-negotiable rule for how that belief must change when new evidence arrives. You start with a prior belief, you observe data, and you compute a posterior belief. Nothing here is unique to statistics — this same rule is the mathematical core of spam filters, medical diagnosis, satellite signal decoding, and modern machine learning classifiers. We will build it from a definition you already know: conditional probability.

Rebuilding conditional probability

For two events A and B in the same sample space, with P(B) > 0, the conditional probability of A given B is defined as:

P(A | B) = P(A ∩ B) / P(B)

Read this as: once we know B has happened, we throw away every outcome outside B and ask what fraction of the remaining space is occupied by A. B becomes the new, smaller universe; P(A | B) measures how much of that smaller universe A occupies. This single idea — restricting the sample space to what you now know is true — is the entire intuition behind Bayesian updating. Every time new evidence arrives, you are doing exactly this: shrinking the universe of possibilities to the ones consistent with the evidence, and re-measuring your hypothesis's share of that shrunken universe.

Deriving Bayes' Theorem, step by step

The definition above can be written two ways, because P(A ∩ B) = P(B ∩ A):

P(A | B) · P(B) = P(A ∩ B) = P(B | A) · P(A)

Divide both sides by P(B), and you have Bayes' Theorem:

P(A | B) = [ P(B | A) · P(A) ] / P(B)

That is the entire derivation — two lines, no hidden steps. What makes it powerful is what each term means when A is a hypothesis and B is observed evidence:

• P(A) — the prior: how likely the hypothesis was before you saw any evidence.
• P(B | A) — the likelihood: how likely this specific evidence is, assuming the hypothesis is true.
• P(B) — the evidence (or marginal probability): how likely this evidence is overall, across every hypothesis.
• P(A | B) — the posterior: your updated belief in the hypothesis, now that you have accounted for the evidence.

When there are only two competing hypotheses, A and its complement A′ (for example: "has the disease" and "does not have the disease"), P(B) itself has to be built up using the law of total probability — B can only happen via A or via A′, and these two paths do not overlap:

P(B) = P(B | A) · P(A) + P(B | A′) · P(A′)

Substituting this into the denominator gives the two-hypothesis form of Bayes' Theorem that you will use constantly:

P(A | B) = [ P(B | A) · P(A) ] / [ P(B | A) · P(A) + P(B | A′) · P(A′) ]

Worked Example 1: the screening test, solved two ways

Let's put real numbers on the opening scenario. Suppose the disease affects 2% of the population (prevalence). The test has 95% sensitivity — P(positive | disease) = 0.95 — and 90% specificity — P(negative | no disease) = 0.90, which means P(positive | no disease) = 0.10 (a 10% false-positive rate). A person tests positive. What is P(disease | positive)?

Method A — natural frequencies (no formula needed). Imagine 1000 random people take this test.

• 2% have the disease: 20 people. 98% do not: 980 people.
• Of the 20 with the disease, 95% test positive: 19 true positives. The remaining 1 is a false negative.
• Of the 980 without the disease, 10% test positive (false-positive rate): 98 false positives. The remaining 882 are true negatives.
• Total people who test positive: 19 + 98 = 117.
• Of those 117 positive results, only 19 actually have the disease.

P(disease | positive) = 19 / 117 ≈ 0.1624, or about 16.2% — nowhere near the 95% most people guess. The test is doing its job (it is genuinely 95% sensitive), but because the disease is rare, the sea of false positives from the enormous healthy population swamps the small number of true positives from the tiny sick population.

Method B — the formula. Using P(D) = 0.02, P(D′) = 0.98, P(+|D) = 0.95, P(+|D′) = 0.10:

P(D | +) = (0.95 × 0.02) / (0.95 × 0.02 + 0.10 × 0.98) = 0.019 / (0.019 + 0.098) = 0.019 / 0.117 ≈ 0.1624

Both methods agree exactly, because Method A is just Bayes' Theorem with the algebra done using whole people instead of decimals. Here is that same computation as a short, general-purpose function, so you can check it for any prevalence and any test accuracy:

def bayes_posterior(prior, sensitivity, false_positive_rate):
    prior_not = 1 - prior
    evidence = sensitivity * prior + false_positive_rate * prior_not
    return (sensitivity * prior) / evidence

result = bayes_posterior(prior=0.02, sensitivity=0.95, false_positive_rate=0.10)
print(round(result, 4))
# 0.1624

Trace it by hand: prior_not = 0.98. evidence = 0.95*0.02 + 0.10*0.98 = 0.019 + 0.098 = 0.117. return 0.019/0.117, which rounds to 0.1624 — matching both methods above.

The diagram below shows the full tree of 1000 people, split first by disease status and then by test result, with the two positive branches feeding into the posterior calculation.

The misconception this example is built to correct

The instinctive answer of "95%" comes from silently swapping two different conditional probabilities: P(positive | disease) — which really is 95%, that's the sensitivity — with P(disease | positive), which is the number you actually want and is only 16.2%. These are not the same quantity, and in general P(A | B) ≠ P(B | A). This mix-up has a name — the base-rate fallacy (also called the prosecutor's fallacy when it shows up in courtrooms: "the chance of this evidence given innocence is tiny, therefore the chance of innocence given this evidence is tiny," which does not follow without accounting for how rare guilt was to begin with). The fix is always the same: never quote a likelihood as if it were a posterior. Ask yourself which population you are the denominator of — everyone who tested positive (117 people), not everyone who has the disease (20 people) — and the fallacy becomes hard to fall for again.

A faster route: the odds form of Bayes' Theorem

Bayes' Theorem has an equivalent, often quicker form in terms of odds. Recall odds(A) = P(A) / P(A′). Dividing the posterior for A by the posterior for A′ cancels the shared denominator P(B):

Posterior odds = Likelihood ratio × Prior odds, where the likelihood ratio is P(B | A) / P(B | A′)

Checking Example 1: prior odds = P(D)/P(D′) = 0.02/0.98 = 1/49. The likelihood ratio = sensitivity / false-positive-rate = 0.95/0.10 = 9.5. Posterior odds = 9.5 × (1/49) ≈ 0.1939. Converting back to a probability: P(D|+) = odds / (1 + odds) = 0.1939 / 1.1939 ≈ 0.1624 — the same 16.2% as before, reached without ever computing P(B) directly. The odds form is especially convenient when you are updating on several independent pieces of evidence in a row, because likelihood ratios simply multiply.

Bayesian inference is a process, not a one-shot formula

The real strength of the Bayesian framework only shows up once you see it used repeatedly — belief updated after each new piece of evidence, with today's posterior becoming tomorrow's prior. This is literally what "learning from data" means in this chapter's title.

Suppose you are handed a coin and told it is either a fair coin (P(heads) = 0.5) or a biased coin that favours heads (P(heads) = 0.75). You have no reason to prefer one hypothesis over the other, so your prior is 50-50. You flip it three times and see: Heads, Heads, Tails. After each flip, update your belief using that flip's outcome as the evidence and last flip's posterior as this flip's prior.

Notice the two directions of movement. Each Heads pushes belief toward "biased" (heads is more likely under that hypothesis, so it's evidence in its favour). The Tails pulls belief back toward "fair," because Tails is relatively more likely under the fair hypothesis (probability 1/2) than under the biased one (probability 1/4). After three flips the biased hypothesis is still favoured, but only barely — 9/17 versus 8/17 — which is exactly what you should expect from a 2-heads-1-tail sample: weak evidence, weakly held belief. Collect ten more flips and the posterior will swing decisively toward whichever hypothesis actually matches the coin. This is the essence of Bayesian inference: beliefs are numbers between 0 and 1, and every observation nudges those numbers by a precise, derivable amount — never by gut feeling.

Worked Example 2 (competitive-exam style): three sources, one observation

This structure — several possible "sources," each with a known probability of producing the observed outcome — is the most common Bayes' Theorem archetype in JEE Main, BITSAT, and olympiad-style questions. Work through it once carefully and the pattern becomes reusable.

A factory has three machines, A, B, and C, producing 25%, 35%, and 40% of its total output respectively. From past records, machine A produces defective items 5% of the time, B does so 4% of the time, and C does so 2% of the time. An item is picked at random from the factory's total output and found to be defective. What is the probability it came from machine A?

Let A, B, C be the events "item came from machine A / B / C," and let X be the event "item is defective." Given: P(A) = 0.25, P(B) = 0.35, P(C) = 0.40, and P(X|A) = 0.05, P(X|B) = 0.04, P(X|C) = 0.02.

First, the total probability of picking a defective item, summed across all three sources:

P(X) = P(X|A)P(A) + P(X|B)P(B) + P(X|C)P(C) = (0.05)(0.25) + (0.04)(0.35) + (0.02)(0.40) = 0.0125 + 0.0140 + 0.0080 = 0.0345

Now apply Bayes' Theorem for the specific source A:

P(A | X) = P(X|A)·P(A) / P(X) = 0.0125 / 0.0345 ≈ 0.3623

So even though machine A only makes 25% of the output, it accounts for about 36.2% of the defective items — because it is by far the least reliable machine. This is the same pattern as the screening-test example, generalised from two hypotheses to three: build every hypothesis's contribution to the evidence, sum them for the denominator, then take the one hypothesis you care about as the numerator.

Where this shows up in your exams

Conditional probability and Bayes' Theorem form a core part of the Probability unit in CBSE Class 12 Mathematics, and the underlying reasoning is assumed background for Class 10-12 statistics and data-handling questions generally. In JEE Main and JEE Advanced, "total probability + Bayes' Theorem" word problems (bags of coloured balls, factories with multiple machines, multiple diagnostic tests) are a recurring and high-scoring question type — the three-machine problem above is a direct template. BITSAT tests the identical skill under tighter time pressure, which is exactly why the natural-frequency method (Method A above) is worth practicing: it is faster and far less error-prone under exam conditions than plugging into the formula. Olympiad-style problem sets go further and combine Bayesian updating with combinatorics or expectation. And if you continue toward a computer science or data science path, this is the same theorem that underlies the "Probability and Statistics" foundations expected before GATE-level machine learning topics — spam filters, medical-imaging classifiers, and recommendation systems all run Bayes' Theorem, at scale, on every prediction they make.

Summary

• Bayesian inference treats probability as a degree of belief that updates in a precise, derivable way when new evidence arrives.
• Bayes' Theorem falls directly out of the definition of conditional probability: P(A|B) = P(B|A)·P(A) / P(B), with the denominator built from the law of total probability when there are multiple hypotheses.
• The four named quantities are prior P(A), likelihood P(B|A), evidence P(B), and posterior P(A|B) — learn these names, exam questions use them.
• For a rare condition, even a highly accurate test can produce a posterior far below its sensitivity, because false positives from the large healthy population outnumber true positives from the small affected population. Natural-frequency reasoning (pretend a round number of people, e.g. 1000) makes this concrete and is usually faster than the raw formula.
• Confusing P(evidence | hypothesis) with P(hypothesis | evidence) is the base-rate fallacy — always check which group you are taking a fraction of.
• The odds form — posterior odds = likelihood ratio × prior odds — is a fast shortcut, especially useful when combining multiple pieces of evidence.
• Bayesian inference is inherently sequential: each posterior becomes the next prior, so belief can be updated one observation at a time, converging toward the truth as evidence accumulates.
• This exact reasoning is tested across CBSE Class 12 boards, JEE Main/Advanced, BITSAT, and olympiad-style questions, and underlies real-world systems from spam filters to medical-imaging AI.

Active recall

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where bayesian inference: learning from data is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting bayesian inference: learning from data to at least 3 other topics you have studied.