AI Bias and Fairness: Ensuring Ethical AI Systems

A Model That Never "Sees" Gender or Race Can Still Discriminate

In 2016, journalists at ProPublica examined a risk-assessment tool called COMPAS, used by several U.S. courts to predict whether a defendant would re-offend. The tool never used race as an input. Yet among defendants who were flagged "high risk" but never actually re-offended, Black defendants were labelled high risk at roughly twice the rate of white defendants (around 45% versus 23%). The company that built COMPAS pushed back, pointing out that its predictions were equally accurate for both groups when you measured accuracy a different way. Both sides were using real arithmetic. Both were, in a narrow sense, correct. This chapter exists to teach you exactly what happened there: how a system with no explicit prejudice programmed into it can still produce unequal outcomes, how to measure that inequality precisely instead of by gut feeling, and why two people can look at the same numbers and reach opposite conclusions about whether the system is "fair."

This is not a chapter about programmers being careless or evil. It is a chapter about arithmetic — about what happens when you optimise a model for one number (overall accuracy) while several other numbers, each a reasonable definition of "fairness," move independently of it. By the end, you will be able to take a table of outcomes split by group and calculate, by hand, whether a system is biased, in what specific sense it is biased, and what a genuine fix looks like versus a cosmetic one.

Where Bias Actually Enters a System

An AI model does not invent bias out of nowhere. It learns patterns from data that humans generated, and those patterns can carry forward old inequalities. Three sources matter most:

• Historical bias. If the training data reflects decisions made by biased humans in the past, the model learns to imitate that bias. In 2018, Reuters reported that an internal recruiting tool built by Amazon had been trained on ten years of resumes submitted to the company — a period when the tech industry hired mostly men. The model learned to downgrade resumes containing the word "women's" (as in "women's chess club captain") and to penalise graduates of certain all-women's colleges, because those patterns were statistically associated with rejection in its training history. Amazon scrapped the tool before deploying it.
• Sampling bias. If certain groups are under-represented in the data the model learns from, the model performs worse for them simply because it saw fewer examples. This was the central finding of the 2018 "Gender Shades" study by researcher Joy Buolamwini and Timnit Gebru, which tested commercial face-analysis systems and found error rates for darker-skinned women reaching as high as 34.7%, compared to well under 1% for lighter-skinned men — largely because the benchmark datasets those systems were built and tested on were overwhelmingly composed of lighter-skinned faces.
• Proxy variables. Even when a sensitive attribute like gender, caste, or religion is deliberately excluded from the data, other columns can silently encode it. This is important enough that it deserves its own section below.

Common Misconception: "Just Delete the Sensitive Column"

A very natural first instinct is: if you're worried a model might discriminate by gender, just remove the gender column from the training data. Problem solved — the model literally cannot see gender, so how could it discriminate on that basis?

This does not work, and understanding why is essential. Suppose you're building a model to shortlist loan applicants in India and you remove the "gender" field, but you keep fields like "first name," "employer," "college attended," and "PIN code." First names are strongly correlated with gender in almost every population. PIN codes in Indian cities are frequently correlated with religion or economic class, because housing patterns have historically clustered by community. A college name can correlate with gender if it is a women's college. None of these fields are labelled "gender" or "religion," but a statistical model doesn't need a field to be labelled correctly — it only needs a field that is correlated with the outcome it's trying to predict. If the training data contains historical bias against a group, the model will find whatever combination of "neutral-looking" columns lets it reconstruct that group membership and continue the pattern. This is called using a proxy variable, and it is one of the most common ways well-intentioned bias mitigation fails in practice. Genuinely testing for fairness requires measuring outcomes by group directly — which is exactly what the rest of this chapter teaches you to do — not just checking which column names appear in the training data.

A Worked Example: Loan Approvals

Let's build the actual arithmetic of fairness measurement from a concrete, invented example. Imagine a bank has built an AI model to recommend loan approvals. Ten men and ten women apply. In reality — based on income stability, repayment history, and other legitimate factors — exactly 7 of the 10 men and 7 of the 10 women are genuinely creditworthy (this is deliberately identical for both groups; hold onto that fact, it matters later). Here is what the model actually decided:

• Men: All 7 creditworthy men are approved, plus 1 of the 3 non-creditworthy men is also approved (mistakenly). Total approved: 8 of 10.
• Women: Only 4 of the 7 creditworthy women are approved (the other 3 creditworthy women are wrongly rejected); none of the 3 non-creditworthy women are approved. Total approved: 4 of 10.

The first and simplest fairness question you can ask is: what fraction of each group got approved? This is called the selection rate, and it's the basis of a fairness definition called demographic parity (a model satisfies demographic parity if the selection rate is equal across groups):

Selection rate (men)   = 8 / 10 = 80%
Selection rate (women) = 4 / 10 = 40%

An 80% vs 40% gap is large and should immediately raise a question. But notice something important: the underlying qualification rate was identical for both groups — 7 out of 10, or 70%, for men and for women. The applicant pools were equally good. The gap in outcomes is entirely a product of how the model treated each group, which is exactly the kind of evidence that should make you suspect a biased process rather than a biased population.

Looking Deeper: Two Different Kinds of Mistake

Selection rate alone doesn't tell you what kind of mistake is happening. A model could have a low selection rate for a group either because it's correctly rejecting more unqualified people in that group, or because it's wrongly rejecting qualified people. To tell these apart, we split each group's outcomes into two separate rates, both computed only among people who share the same ground truth:

• True Positive Rate (TPR) — of the people who were actually creditworthy, what fraction did the model correctly approve? A high TPR means the model isn't leaving qualified people behind.
• False Positive Rate (FPR) — of the people who were actually not creditworthy, what fraction did the model mistakenly approve? A high FPR means the model is being too generous with unqualified applicants.

Let's compute both for our example:

Men:
  TPR = (creditworthy men approved) / (creditworthy men) = 7 / 7 = 100%
  FPR = (non-creditworthy men approved) / (non-creditworthy men) = 1 / 3 ≈ 33%

Women:
  TPR = (creditworthy women approved) / (creditworthy women) = 4 / 7 ≈ 57%
  FPR = (non-creditworthy women approved) / (non-creditworthy women) = 0 / 3 = 0%

Now the picture is much sharper. Men enjoy a perfect 100% TPR — every single creditworthy man got his loan — while women only got a 57% TPR, meaning 3 out of 7 creditworthy women were wrongly turned away. Meanwhile the model's FPR for women is a spotless 0% (it never approved an unqualified woman) while for men it let one unqualified applicant through. In plain language: the model is being unnecessarily strict with women (rejecting qualified applicants it shouldn't) and slightly too lenient with men (approving one it shouldn't have). A fairness definition built on equalising TPR across groups is called equal opportunity — and by that definition, this model is clearly failing.

Two Very Different Ways to "Fix" the 40-Point Gap

Suppose the bank's leadership sees the 80%-vs-40% selection rate gap and orders it fixed. There are at least two ways to close it, and they are not equally good — this is the point in the chapter where you have to think past the headline number.

• Fix A — a quota. The bank randomly approves 4 more women, picked without checking creditworthiness, to bring the women's selection rate up to 80%. The demographic-parity gap disappears. But the 3 women who were wrongly rejected because the model misjudged them are not necessarily among the 4 randomly chosen — some genuinely unqualified women may now get loans they can't repay, and some of the originally wronged, qualified women may still be shut out. The selection-rate number looks fixed. The actual injustice (creditworthy women being disbelieved) may not be.
• Fix B — auditing the rejections. The bank re-examines the 3 creditworthy women who were rejected, finds the model's error, and approves them specifically. Now women's TPR rises from 57% to 100%, matching men's, for a principled reason: previously undetected qualified applicants are now correctly identified. The women's selection rate rises to 70% (7/10) — not identical to men's 80%, because men's FPR of 33% is itself a separate flaw the bank should fix by tightening approvals for unqualified men, not by loosening approvals for women.

Both fixes can produce an identical-looking selection-rate number. Only one of them actually corrects the underlying error. This is why a responsible fairness audit always asks for TPR and FPR by group, not just the raw selection rate — the single number can hide exactly how a gap is being closed.

When Fairness Definitions Truly Conflict

In the loan example above, both groups had an identical 70% true qualification rate. That detail matters a lot, because when two groups have the same underlying base rate, a sufficiently good model can, in principle, satisfy every fairness definition we've discussed simultaneously — a perfect model would approve exactly the 7 creditworthy people in each group, giving both groups a 70% selection rate, a 100% TPR, and a 0% FPR at once. So the unfairness we found in the loan example was not mathematically unavoidable — it was evidence of a correctable, biased process.

Now consider a case where the two groups genuinely differ in their true qualification rate. Two bank branches, District X and District Y, each have 10 applicants. In District X, 8 of the 10 are genuinely creditworthy. In District Y, only 2 of the 10 are. Suppose we had an ideal, perfectly accurate model — one that approves exactly the creditworthy applicants in each district and nobody else:

District X: approves 8 of 10 → selection rate 80%, TPR = 8/8 = 100%, FPR = 0/2 = 0%
District Y: approves 2 of 10 → selection rate 20%, TPR = 2/2 = 100%, FPR = 0/8 = 0%

Both districts get identical, perfect error rates — 100% TPR and 0% FPR in both. By the "equal opportunity" definition, this model is completely fair. But the selection rates are wildly different, 80% vs 20% — this same perfect model badly fails demographic parity, simply because the two districts have different underlying qualification rates. There is no way to fix this by improving the model's accuracy, because the model is already perfect.

Now watch what happens if a regulator instead insists on demographic parity — say, both districts must have exactly a 50% selection rate:

District X: must approve 5 of 10. Best case, all 5 come from the 8 creditworthy → TPR = 5/8 = 62.5%, FPR = 0/2 = 0%
District Y: must approve 5 of 10, but only 2 are creditworthy. Best case: approve those 2, plus 3 more from the 8 non-creditworthy →
            TPR = 2/2 = 100%, FPR = 3/8 = 37.5%

Now the selection rates match, but TPR and FPR diverge sharply between districts — District Y is forced to approve unqualified applicants (37.5% FPR) just to hit the parity target, something District X never has to do. This is the real, general phenomenon described by researchers Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan (2016) and Alexandra Chouldechova (2017): whenever two groups have genuinely different base rates, no model — however accurate — can generally satisfy demographic parity and equalised error rates (equal TPR and FPR) at the same time. You can pick which fairness definition to prioritise, but you usually cannot have all of them when the groups' underlying rates truly differ. This, precisely, was the mathematical crux of the COMPAS debate from the start of this chapter: Black and white defendants in the dataset had different underlying reoffense rates, so ProPublica's finding of unequal false-positive rates and the company's defence of equal predictive accuracy were, mathematically, both real, simultaneous, unavoidable consequences of that base-rate gap — not a case of one side simply being wrong.

Measuring Fairness in Code

The arithmetic above is exactly what a fairness audit does in practice, just automated. Here is the loan example, computed in Python using plain loops so every step is traceable by hand:

# Each applicant record: (gender, is_creditworthy, was_approved)
# Built directly from our example's numbers:
#   Men:   7 creditworthy (all approved) + 3 not creditworthy (1 approved)
#   Women: 7 creditworthy (4 approved)   + 3 not creditworthy (0 approved)

applicants = []

# --- Men ---
for i in range(7):
    applicants.append(("man", True, True))      # creditworthy man, approved
for i in range(3):
    approved = (i == 0)                         # only the 1st of these 3 is approved
    applicants.append(("man", False, approved))

# --- Women ---
for i in range(7):
    approved = (i < 4)                          # only the first 4 of these 7 approved
    applicants.append(("woman", True, approved))
for i in range(3):
    applicants.append(("woman", False, False))  # none of these 3 approved

# Compute selection rate, TPR, and FPR for each gender using plain counters
for gender in ["man", "woman"]:
    total = 0
    approved_count = 0
    creditworthy_count = 0
    true_positive = 0     # creditworthy AND approved
    false_positive = 0    # NOT creditworthy BUT approved

    for record in applicants:
        g, creditworthy, approved = record
        if g != gender:
            continue
        total = total + 1
        if approved:
            approved_count = approved_count + 1
        if creditworthy:
            creditworthy_count = creditworthy_count + 1
            if approved:
                true_positive = true_positive + 1
        else:
            if approved:
                false_positive = false_positive + 1

    not_creditworthy_count = total - creditworthy_count
    selection_rate = approved_count / total * 100
    tpr = true_positive / creditworthy_count * 100
    fpr = false_positive / not_creditworthy_count * 100

    print(f"{gender}: selection={selection_rate:.0f}%  TPR={tpr:.0f}%  FPR={fpr:.0f}%")

# Output:
# man: selection=80%  TPR=100%  FPR=33%
# woman: selection=40%  TPR=57%  FPR=0%

Trace it yourself: the men loop adds 7 records of ("man", True, True), then 3 records where only i == 0 is approved — so 1 approved, 2 rejected. That gives men 10 total, 8 approved, 7 creditworthy, 7 true positives, 1 false positive — exactly matching the hand-computed 80% / 100% / 33% above. The women loop does the same with i < 4 for the creditworthy group (4 approved, 3 rejected) and all three non-creditworthy women rejected, giving 40% / 57% / 0%. Notice the code never touches a "fairness library" — it is the same three ratios, computed group by group, every time.

Building Fairer Systems in Practice

Since you now know that no single number tells the whole story, a real fairness audit checks several things together, not just one:

• Audit outcomes by group before deployment — compute selection rate, TPR, and FPR separately for every relevant group, the same way you did above, rather than trusting one overall accuracy figure.
• Check for proxy variables — even after removing a sensitive column, test whether the model's decisions still correlate strongly with group membership using held-out data where you do know the sensitive attribute (for measurement purposes only, kept separate from the model's inputs).
• Decide which fairness definition matters most for the specific decision — a medical screening test might prioritise equal TPR (catching the disease in everyone who has it) even at the cost of unequal FPR, while a scholarship-selection tool might prioritise demographic parity. There is rarely one universally "correct" choice; it depends on the real-world cost of each type of error.
• Prefer targeted correction over blanket quotas — as Fix B in our loan example showed, fixing the specific errors a model makes is more defensible and more effective than adjusting numbers until they merely look balanced.

Check Your Understanding

1. A hospital's AI triage tool has a 90% overall accuracy rate. A doctor claims this proves the tool is fair to all patient groups. Explain, using the concepts of selection rate, TPR, and FPR, why overall accuracy alone cannot prove this.
2. A company removes the "caste" and "religion" columns from its hiring-model training data and announces the model is now "bias-free." Using the idea of a proxy variable, explain why this claim can still be false, and give one example of a column that might act as a proxy in an Indian hiring context.
3. In the two-district example, District X has 8 of 10 applicants genuinely creditworthy and District Y has 2 of 10. (a) Calculate each district's selection rate under a perfect model that approves exactly the creditworthy applicants. (b) Explain why both districts get identical TPR and FPR (100% and 0%) under this perfect model even though their selection rates differ so much. (c) Now suppose a regulator instead forces both districts to have exactly a 50% selection rate. Recompute the best-case TPR and FPR for each district under this rule, and explain what happened to the equal-error-rate fairness that the perfect model had achieved.
4. In the loan example, Fix A (a random quota of extra approvals) and Fix B (auditing and correcting specific wrongful rejections) both raise the women's selection rate, but only one directly raises women's TPR. Which one, and why does that distinction matter if the real goal is correcting a biased process rather than just balancing a number?
5. Invent your own small example (8 or 10 people per group is enough) where two groups have equal selection rates but different TPRs. Show your numbers and calculations.

Summary

Bias in an AI system is not a vague accusation — it is a measurable gap between how a model treats different groups, and you now have precise tools to find and describe it. Selection rate tells you what fraction of each group is chosen; it underlies demographic parity. True Positive Rate and False Positive Rate, computed separately for each group, tell you what kind of mistake the model is making and to whom; they underlie equal opportunity and equalised odds. Bias enters models mainly through historical patterns baked into training data, under-representation of some groups in that data, and proxy variables that let a model reconstruct a sensitive attribute even after it has been formally removed — which is why deleting a column is never, by itself, proof of fairness. When two groups genuinely have the same underlying qualification rate, a good-enough model can satisfy every fairness definition at once, and any gap you observe is evidence of a fixable, biased process. But when two groups have genuinely different base rates, mathematics itself guarantees that demographic parity and equalised error rates cannot both hold except in special cases — a result documented by Kleinberg, Mullainathan, and Raghavan, and by Chouldechova, and the exact reason the real COMPAS debate had no simple resolution. Your job, as someone who builds or evaluates these systems, is never to accept a single headline number. It is to compute the breakdown, identify which specific definition of fairness is being violated, and choose a correction that fixes the actual error rather than one that merely repaints the summary statistic.