A/B Testing for Machine Learning: Evaluating in Production

The problem: two models, one decision

Imagine you are on the machine learning team at an Indian ed-tech app that recommends the next practice question to a student after every attempt. The current model — call it Model A — has been running for a year. A new team member has just finished training Model B, a redesigned recommender that, when tested on last month's saved data, scores higher on the standard evaluation metric than Model A does on that same saved data. The obvious move seems to be: replace Model A with Model B for all ten million users tonight.

This is exactly the moment where most ML systems in the real world go wrong, and it is the reason this chapter exists. A model's score on a fixed, saved dataset — what engineers call an offline evaluation — tells you how well it reproduces past decisions. It does not tell you how real students, clicking in real time, will actually respond when the model's suggestions change what they see next. Offline data is frozen; production is alive. Students react to what the model shows them, and that reaction is exactly what the offline test could never capture, because in the offline data nobody was reacting to Model B — Model B did not exist yet when that data was collected.

So the real question a production ML team must answer is not "which model scores higher on old data?" but "which model produces better outcomes when it is actually running, in front of actual people, today?" Answering that question correctly, with real users, without wrecking their experience if the new model turns out to be worse, is what A/B testing is for.

A simpler starting point: comparing two study strategies

Before we touch a single line of ML code, let's strip the idea down to something you can reason about with nothing but arithmetic. Suppose your school wants to know whether a new revision method (call it Method B — solving mixed practice papers) helps students score better on a mock CBSE test than the old method (Method A — chapter-wise revision). Here is the wrong way to test it: let the toppers of Class 9-A choose Method B because they're curious, while the rest of the class keeps using Method A, and then compare the two groups' scores. Even if Method B "wins," you have learned nothing about the method — you have only rediscovered that toppers score higher than average, which was true before the experiment even started.

The right way is to take one pool of otherwise similar students and randomly split them into two groups — say by tossing a coin for each student — so that neither group is systematically stronger or weaker to begin with. One group revises with Method A, the other with Method B, everything else about their week held as similar as possible, and only then do you compare mock-test scores. Random assignment is what makes the comparison fair: any difference in scores can now be credited to the method itself, not to who happened to end up in which group.

This is the entire logic of an A/B test. The "coin toss" is randomization. The untouched group is the control group. The group that gets the new thing is the treatment group. And the score you compare at the end is the metric. Everything that follows in this chapter is this same four-part idea, applied to software that serves millions of requests a second instead of thirty students in a classroom.

From study groups to production traffic

Back to the ed-tech app. Instead of splitting a classroom, you split incoming users. When a student opens the app, a piece of code decides, before showing anything, whether that student's session belongs to the control group (sees Model A, the existing recommender) or the treatment group (sees Model B, the new one). The split has to satisfy the same fairness requirement as the classroom experiment: it must be random with respect to anything that could affect the outcome — you cannot let students who use the app more actively self-select into one group, and you cannot let the app team's Bengaluru interns manually pick "engaged" users for the new model to make it look good.

The metric also has to be chosen with the same care as "mock test score" was chosen for the study-method experiment. For a recommender, a natural metric is click-through rate (CTR) — of the students who were shown a recommended question, what fraction actually clicked on it and attempted it? A good production metric is something you can measure automatically, for every user, without asking anyone how they feel — the app already logs every click, so CTR costs nothing extra to compute.

Worked example: comparing click-through rates

Suppose the app runs this test for a week. 5,000 students are randomly routed to the control group and see Model A's recommendations; another 5,000, chosen the same random way, see Model B's. At the end of the week, the logs show:

• Model A (control): 620 clicks out of 5,000 students shown a recommendation.
• Model B (treatment): 690 clicks out of 5,000 students shown a recommendation.

The click-through rate for each group is just clicks divided by users shown, turned into a percentage:

clicks_A, users_A = 620, 5000
clicks_B, users_B = 690, 5000

ctr_A = clicks_A / users_A * 100
ctr_B = clicks_B / users_B * 100

print(f"Model A CTR: {ctr_A:.1f}%")
print(f"Model B CTR: {ctr_B:.1f}%")
print(f"Lift: {ctr_B - ctr_A:.1f} percentage points")
print(f"Relative lift: {(ctr_B - ctr_A) / ctr_A * 100:.1f}%")

Tracing this by hand: ctr_A = 620 / 5000 * 100 = 12.4 and ctr_B = 690 / 5000 * 100 = 13.8. The program prints:

Model A CTR: 12.4%
Model B CTR: 13.8%
Lift: 1.4 percentage points
Relative lift: 11.3%

Model B's CTR is 1.4 percentage points higher in absolute terms, and that 1.4-point gain is about 11.3% larger relative to where Model A started (1.4 ÷ 12.4 ≈ 0.113). On the surface, this looks like a clear win for Model B. But before you roll it out to every student in the country, you need to ask one more question — the question that separates a careful ML engineer from a careless one.

Why sample size matters: the wobble problem

Here is the same experiment, run with far fewer students — 50 in each group instead of 5,000:

• Model A: 6 clicks out of 50 students → 6/50 = 12%.
• Model B: 7 clicks out of 50 students → 7/50 = 14%.

Notice this is almost the identical CTR gap as before (roughly 12% vs 14%), yet you should trust this second result far less. Why? Because with only 50 students, a single extra click swings the percentage by two whole points. If the "true," long-run click rate for Model A is really 12%, then just by ordinary random luck, a batch of 50 students could easily produce 4 clicks (8%), or 9 clicks (18%), purely because 50 people is a small sample and human behaviour has natural variation from batch to batch — some batches happen to contain more curious students, some fewer, with no model change involved at all.

You can see this "wobble" precisely using the same idea as flipping a coin: if a fair coin's true chance of heads is 50%, flipping it only 10 times might easily give you 7 heads (70%) purely by chance, even though nothing about the coin changed. Flip it 10,000 times, though, and the fraction of heads will almost always land close to 50%, because the individual flukes of luck average out over a large number of trials. The same thing happens with click rates: a small sample can produce a big-looking gap purely from randomness, while a large sample lets real differences show through the noise.

This is precisely why the 5,000-user version of our experiment is trustworthy in a way the 50-user version is not. With 5,000 students per group, an outcome as skewed as "690 clicks instead of the roughly 620 you'd expect if nothing changed" is much harder to explain away as pure luck — there are simply too many independent students involved for chance alone to consistently push the numbers in one direction. Data scientists formalize this "is the gap bigger than we'd expect from luck alone?" question using statistical significance tests, which you will meet formally in later statistics coursework; the essential intuition to take from Grade 9, though, is this: a bigger sample size shrinks the amount a metric can wobble from chance alone, so only compare A/B results after checking the group sizes are large enough that the gap you're looking at isn't just noise. Production ML teams generally will not ship a model based on a gap seen in a few hundred users — they wait until enough traffic has flowed through both groups that the difference is unlikely to have happened by chance.

Common misconception: "the model with higher offline accuracy will always win the A/B test"

It is tempting to believe that if Model B already beat Model A on the saved offline test set, the A/B test is just a formality — a real-world confirmation of a result you already know. This is false, and it is one of the most expensive mistakes in production ML. Offline accuracy is measured against past user behaviour that was itself generated while an older model (or no model) was running. The moment a new model changes what users are shown, users change how they behave in response — a phenomenon sometimes called a feedback loop. A recommender that scores brilliantly at predicting what students clicked on last year, when a completely different model was choosing what to show them, can perform worse in production because it recommends questions in a pattern real students find repetitive or frustrating in ways the offline metric never measured. Offline evaluation and production A/B testing measure genuinely different things: one measures how well a model reconstructs old decisions; the other measures what happens when the model is the one making new decisions. Both are necessary — offline tests are cheap, fast filters to catch obviously broken models before they ever reach a real user — but only the A/B test tells you the thing you actually care about, which is real-world impact. Never skip the A/B test just because the offline number looked good.

Designing a fair test: guardrails and pitfalls

Running an honest A/B test requires more discipline than just splitting users and comparing one metric. A few things a careful test must get right:

• Consistent assignment. A student who opens the app three times during the test week must land in the same group every time. If a student saw Model A on Monday and Model B on Wednesday, you can no longer cleanly attribute their behaviour to either model, and your comparison is contaminated.
• One primary metric, chosen before the test starts. If you check ten different metrics after the results come in and report only the one that happened to favour Model B, you are very likely reporting noise, not a real effect — with enough metrics, some will look "significant" purely by chance. Decide the metric that matters (here, CTR) before you look at the data.
• Guardrail metrics. Even if CTR goes up, you must check that other important numbers — app crashes, page-load time, how long students stay in the app — do not silently get worse. A model that raises clicks by making recommendations look more clickbait-y, while quietly increasing how often students quit the app in frustration, is not actually an improvement.
• The novelty effect. Users sometimes click more on anything that looks different, simply because it's new and unfamiliar — not because it's genuinely better. A one-week test can be fooled by this. Longer tests, or repeating the test after the novelty has faded, help separate a real improvement from simple curiosity.
• Sample ratio mismatch. If the test is supposed to split 50/50 but the logs actually show 5,000 users in control and only 3,200 in treatment, something is broken in the assignment code itself (perhaps Model B's app path crashes for some users before they can even be logged), and the whole test's results become untrustworthy until that bug is found and fixed.

Building it in code: consistent random assignment

The "consistent assignment" requirement above rules out something naive like generating a fresh random number every time a student opens the app — that would flip them between groups on every visit. Instead, engineers derive the group from something stable about the user — their user ID — run through a hash function, which turns any input into a fixed-size number in a way that looks random but is completely deterministic: the same input always produces the same output.

import hashlib

def assign_group(user_id, salt="ab_test_2026"):
    digest = hashlib.md5(f"{salt}_{user_id}".encode()).hexdigest()
    bucket = int(digest, 16) % 100
    return "A" if bucket < 50 else "B"

for uid in ["user_101", "user_205", "user_309", "user_101"]:
    print(uid, "->", assign_group(uid))

Tracing this: hashlib.md5(...) converts the salted user ID into a long hexadecimal string (the "digest"); int(digest, 16) reads that hex string as one huge integer; % 100 squeezes that huge, effectively-random-looking integer down to a bucket number from 0 to 99; and buckets below 50 go to group A, the rest to group B, giving an even 50/50 split across many users. Running this exact code produces:

user_101 -> B
user_205 -> A
user_309 -> A
user_101 -> B

Notice user_101 appears twice in the list and gets group B both times — that's the whole point. Because MD5 is deterministic, the same user ID with the same salt always lands in the same bucket, so a student sees a consistent version of the app across every visit, every day, for the whole duration of the test. The salt string exists so that if you later run a completely different A/B test on the same users (say, testing a new UI colour scheme), you can change the salt to get an independent, uncorrelated split rather than always splitting the exact same students the exact same way.

Beyond simple A/B: A/A tests and staged rollouts

Two techniques experienced ML teams use to trust their A/B testing pipeline even more: An A/A test means running the split-and-compare machinery while showing both groups the identical Model A. Since nothing actually differs between the groups, any metric gap you observe must be pure noise from random sampling — exactly like the coin flip idea from earlier. If an A/A test regularly reports a "significant" difference where none should exist, that's a red flag that the testing pipeline itself — the randomization code, the logging, the metric calculation — has a bug, and it must be fixed before any real A/B result from that pipeline can be trusted.

A staged rollout (sometimes called a canary release) is the practical, risk-managed version of an A/B test: instead of jumping straight to a 50/50 split of ten million users, a new model is first shown to a tiny slice — say 1% of traffic — while engineers watch the guardrail metrics closely. If nothing breaks, the treatment slice is grown to 5%, then 20%, then 50%, with the option to instantly roll back to 0% at any stage if something goes wrong. This way, if Model B turns out to have a serious flaw the offline tests missed, only a small fraction of real users are ever affected by it, rather than the whole user base finding out at once.

Where this fits in your AI/CS learning

The CBSE AI curriculum's project cycle — problem scoping, data acquisition, data exploration, modelling, and evaluation — usually introduces evaluation through metrics like accuracy, precision, and recall computed on a fixed test set, which is exactly the offline evaluation this chapter contrasted with production testing. A/B testing is the natural next stage of that same evaluation idea: once a model leaves the lab and starts making live decisions for real users, "how does it score on my saved test data" stops being the only question that matters, and "how does it perform for real people, right now, compared to a fair control group" takes over. Understanding both halves — offline metrics to filter out clearly bad models cheaply, and A/B tests to confirm real-world impact before a full rollout — is what separates evaluating a model as a classroom exercise from deploying one responsibly.

Check your understanding

1. An app engineer says: "Model B scored 91% offline accuracy versus Model A's 88%, so I'm switching all users to Model B tonight, no A/B test needed." Explain precisely what could go wrong with this plan.
2. A weekend A/B test shows Model A with a CTR of 10/80 = 12.5% and Model B with a CTR of 14/80 = 17.5%. A colleague says this proves Model B is better. Using the "wobble" idea from this chapter, explain why you should be cautious about this conclusion, and describe what you would do differently to get a trustworthy answer.
3. Why must a hash-based group assignment function be deterministic (always give the same user the same group) rather than randomly re-rolled on every app open? What specific problem would non-deterministic assignment cause for the experiment's results?
4. A team runs an A/B test and only reports that "engagement time" went up for Model B, without mentioning crash rate or app-quit rate. What kind of metric is missing from their report, and why is it dangerous to skip it?
5. Design, in words, an A/A test for the ed-tech recommender described in this chapter. What specific result from that A/A test would tell you the testing pipeline has a bug?

Summary

A/B testing answers a question offline evaluation cannot: not "which model reproduces past decisions better," but "which model produces better real-world outcomes when it is actually deciding what real users see." The method starts from the same logic as any fair experiment — randomly split a population into a control group and a treatment group so the only systematic difference between them is the thing being tested, choose one primary metric in advance, and measure it consistently. A larger sample size shrinks how much a metric can "wobble" purely from chance, which is why a gap seen across a handful of users deserves far less trust than the same-sized gap seen across thousands. Consistent, hash-based assignment keeps each user in the same group throughout the test; guardrail metrics catch improvements in one number that hide damage to another; A/A tests validate that the testing pipeline itself is trustworthy; and staged rollouts limit the damage if a new model turns out worse than expected. The single idea worth carrying forward is this: a model's score on saved data is a cheap first filter, never the final word — real deployment decisions are earned through a fair, sufficiently large, carefully guarded comparison against real users in production.