AI Ethics and Bias: The Hard Problems

The Hiring Algorithm That Learned to Reject Women

In 2014, Amazon's machine learning team built an experimental tool to screen job applicants. It read resumes and gave each candidate a score from one to five stars, the same way shoppers rate products on the site. The idea was simple: feed the system ten years of resumes the company had received, along with which candidates were eventually hired, and let it learn what a strong resume looks like. By 2015, engineers noticed the tool was not rating candidates for technical jobs in a gender-neutral way. It was systematically downgrading resumes that contained the word "women's" — as in "women's chess club captain" — and it had taught itself to penalize graduates of two all-women's colleges. Amazon edited the program to stop it from treating those specific terms as negative, but engineers could not be sure the model had not found other, subtler proxies for the same pattern, and the project was scrapped by 2017. This was reported by Reuters in 2018 and remains one of the most carefully documented cases of algorithmic bias in industry.

Notice what did not happen here. Nobody at Amazon wrote a rule saying "reject women." No engineer typed gender into the model as an input. The bias was not injected — it was learned, and it was learned correctly, in the narrow sense that the model was doing exactly what it was trained to do: find the patterns that predicted who got hired in the past. The problem is that the past was not fair, and a model trained to imitate the past will imitate its unfairness with mathematical precision. This is the central fact this chapter is built around: bias in AI is usually not a bug where the code is "wrong." It is the system working exactly as designed, on data that encodes a history you did not intend to teach it.

What "Bias" Means in a Machine Learning System

In everyday speech, "bias" means a person's unfair preference. In machine learning it is a more precise, technical idea: bias is any systematic difference between what a model predicts and what would be a fair or accurate outcome, where the difference is not random noise but a repeatable pattern tied to some attribute like gender, caste, religion, region, or economic background. Researchers who study this (notably Harini Suresh and John Guttag, in a widely cited 2021 framework) break the sources down into distinct stages of the machine learning pipeline, because "fix the bias" is meaningless until you know which stage produced it.

• Historical bias. The world the data was collected from was already unequal, so even a perfectly accurate, perfectly measured dataset reproduces that inequality. Amazon's resumes were historically real; tech hiring really had been skewed male for a decade, and the model correctly detected that pattern.
• Representation bias. The training data under-samples some group, so the model simply has less signal to learn from about them and performs worse on them. A face dataset built mostly from North American and European celebrity photos under-represents darker skin tones, so the model has fewer examples to learn those faces from.
• Measurement bias. The label you train on is a flawed stand-in for what you actually care about. "Was this person arrested again?" (used in criminal risk tools) is not the same as "did this person actually reoffend?" — arrest rates are affected by how heavily different neighbourhoods are policed, not just by behaviour.
• Aggregation bias. One model is forced to fit all groups with a single set of parameters, when the true relationship between features and outcome is genuinely different across groups, so the "average" fit is wrong for everyone.
• Evaluation and feedback-loop bias. The system is tested on a benchmark that itself under-represents some group, so poor performance for that group never even shows up in the reported accuracy — and if the system's own decisions feed back into future training data (for example, a policing model that sends more patrols to a neighbourhood, which then generates more recorded arrests there, which "confirms" the model's prediction), the bias compounds over time instead of being corrected.

Keep this taxonomy in mind, because the rest of this chapter is really about one specific, mathematically precise instance of it: what happens when a model's errors themselves are distributed unequally across groups, even when the model was never told which group a person belonged to.

Reading a Model's Mistakes: The Confusion Matrix, Split by Group

Every binary classifier — a model that outputs "yes" or "no," "high risk" or "low risk," "approve" or "reject" — makes two kinds of mistakes. It can flag someone as positive when they are actually negative (a false positive), or miss someone who is actually positive (a false negative). For a group of people with a true outcome, we count four numbers: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). From these four counts we define the metrics that matter for fairness:

• False Positive Rate, FPR = FP / (FP + TN) — of the people who are actually negative, what fraction did the model wrongly flag?
• False Negative Rate, FNR = FN / (FN + TP) — of the people who are actually positive, what fraction did the model wrongly clear?
• True Positive Rate / Recall, TPR = TP / (TP + FN) = 1 − FNR
• Positive Predictive Value / Precision, PPV = TP / (TP + FP) — of the people the model flagged, what fraction were actually positive?

An "accurate" model, in the sense of overall percentage correct, can still have wildly different FPR or PPV for two different groups, and overall accuracy will not reveal this at all, because accuracy just averages everyone together. This is exactly the gap that turned out to matter in the most influential real-world audit of an algorithmic risk tool.

Case Study: COMPAS and the ProPublica Investigation

COMPAS is a risk-assessment tool used in parts of the United States criminal justice system to score defendants on their likelihood of reoffending, a score judges can see when setting bail or sentencing. In 2016, journalists at ProPublica obtained records for over 7,000 defendants in Broward County, Florida, and compared COMPAS's predictions to whether those individuals actually went on to be arrested again within two years. The tool's overall accuracy was similar across racial groups, at roughly 60 to 65 percent — on the surface, "fair." But when ProPublica split the errors by group, a sharp asymmetry appeared: among defendants who did not reoffend, Black defendants were flagged as high-risk at roughly twice the rate of white defendants (a false positive rate of around 45 percent versus around 24 percent). Among defendants who did reoffend, white defendants were far more likely to have been wrongly cleared as low-risk (a false negative rate of around 48 percent versus around 28 percent for Black defendants). Two groups, similar overall accuracy, opposite error profiles: one group disproportionately over-flagged, the other disproportionately under-flagged.

Northpointe, the company that built COMPAS, responded that its tool was fair by a different, equally legitimate-sounding standard: calibration. Among defendants the tool scored as high-risk, roughly the same fraction actually reoffended, regardless of race — the score meant the same thing for everyone it was given to. Both sides were using real, correctly computed statistics from the same dataset. Both were right about the number they were citing. This was not a data error or a coding bug on either side. As the next section shows, this disagreement was mathematically guaranteed to happen.

Four Definitions of "Fair" — and Why They Cannot All Agree

Before you can even ask whether a model is fair, you have to pick which of several genuinely different, individually reasonable mathematical definitions of fairness you mean.

• Demographic parity (statistical parity): the fraction of people flagged positive is the same across groups — P(Ŷ=1 | Group A) = P(Ŷ=1 | Group B). This asks: does the model select people from each group at the same rate?
• Equal opportunity: among people who are actually positive, the model catches the same fraction in each group — TPR is equal across groups (equivalently, FNR is equal). This asks: if you deserve to be flagged, are your chances of actually being flagged the same regardless of group?
• Equalized odds: both TPR and FPR are equal across groups. This is stricter than equal opportunity — it also demands that innocent people in each group are wrongly flagged at the same rate.
• Predictive parity (calibration): among people the model flags as positive, the same fraction are actually positive in each group — PPV is equal across groups. This asks: does a "high-risk" score mean the same thing no matter who receives it?

Each of these is a legitimate, defensible notion of fairness that a reasonable policy-maker could demand. The uncomfortable mathematical fact, proven independently by Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, and separately by Alexandra Chouldechova, both in 2016 to 2017, is that except in special cases, you cannot satisfy calibration and equalized odds at the same time, whenever the two groups have different base rates — that is, whenever the true proportion of positives genuinely differs between the groups. This is not a limitation of any particular algorithm. It is a theorem: no algorithm, however good, can escape it while base rates differ. Let's derive why.

The Impossibility Result, Derived With Real Numbers

Bayes' rule lets us write PPV in terms of a group's base rate p (the true fraction of positives), and the classifier's FNR and FPR:

PPV = [ p · (1 − FNR) ] / [ p · (1 − FNR) + (1 − p) · FPR ]

Look at what this formula says: PPV is a function of three things — the base rate p, the FNR, and the FPR. If we force FNR and FPR to be identical across two groups (equalized odds), PPV can still come out different across those groups, purely because p is different. Let's see it with counts instead of just symbols, which is the more trustworthy way to check any formula.

Take two groups of 1000 people each, with different true base rates: Group A has a base rate of 30 percent (300 people are truly positive, 700 are truly negative). Group B has a base rate of 60 percent (600 truly positive, 400 truly negative). Now apply a single classifier to both groups that happens to have identical error rates on both: FNR = 30 percent, FPR = 20 percent.

Group A: of the 300 true positives, FNR = 30% means 90 are missed (false negatives) and 210 are correctly caught (true positives). Of the 700 true negatives, FPR = 20% means 140 are wrongly flagged (false positives) and 560 are correctly cleared (true negatives). Total flagged as positive = 210 + 140 = 350. So PPV = 210 / 350 = 0.60.

Group B: of the 600 true positives, FNR = 30% means 180 missed and 420 correctly caught. Of the 400 true negatives, FPR = 20% means 80 wrongly flagged and 320 correctly cleared. Total flagged as positive = 420 + 80 = 500. So PPV = 420 / 500 = 0.84.

Same FPR (20%), same FNR (30%), in both groups — equalized odds holds exactly. Yet PPV is 60% in Group A and 84% in Group B. A "high-risk" label means something very different in each group: in Group A, 4 out of every 10 people flagged high-risk were actually never going to reoffend; in Group B, only about 1.6 out of every 10 flagged people were false alarms. This is not a flaw in this particular example — it is forced by algebra whenever the base rates differ and the error rates are equal. The only way to make PPV equal across groups too would be to let FPR or FNR differ across groups, which breaks equalized odds instead. You can verify the arithmetic in code:

def fairness_metrics(TP, FP, FN, TN):
    FPR = FP / (FP + TN)
    FNR = FN / (FN + TP)
    TPR = TP / (TP + FN)
    PPV = TP / (TP + FP)
    return {"FPR": FPR, "FNR": FNR, "TPR": round(TPR, 2), "PPV": round(PPV, 2)}

group_a = fairness_metrics(TP=210, FP=140, FN=90, TN=560)
group_b = fairness_metrics(TP=420, FP=80, FN=180, TN=320)

print("Group A:", group_a)
print("Group B:", group_b)

# Output:
# Group A: {'FPR': 0.2, 'FNR': 0.3, 'TPR': 0.7, 'PPV': 0.6}
# Group B: {'FPR': 0.2, 'FNR': 0.3, 'TPR': 0.7, 'PPV': 0.84}

This is exactly the disagreement between ProPublica (which reported unequal FPR and FNR) and Northpointe (which reported roughly equal PPV): the true reoffense base rate genuinely differed between the groups in their dataset, so it was mathematically impossible for COMPAS to satisfy both notions of fairness simultaneously. Neither party made an arithmetic error. The lesson is not "one side was lying" — it is that "is this algorithm fair?" is an incomplete question until you specify which of several mutually incompatible mathematical definitions of fairness you are demanding, and that choice is an ethical and policy decision, not something a data scientist can settle by writing better code.

Common Misconception: "Just Delete the Sensitive Column"

A very natural first instinct, once you learn that a model is biased by gender, caste, religion, or race, is to simply remove that column from the training data. Surely a model that never sees "gender" cannot discriminate by gender? This approach is called fairness through unawareness, and it is one of the most persistently tempting wrong answers in this field, because it feels rigorous while doing almost nothing.

It fails because many other features act as proxies — variables that are statistically correlated with the sensitive attribute even though they are not literally that attribute. A postal code can be a strong proxy for caste or religious composition of a neighbourhood, because housing patterns in many Indian cities and towns have historically clustered along community lines. A person's given name can be a strong proxy for perceived religion or region. School attended can be a strong proxy for gender, if it was a single-sex institution. Removing the sensitive column does not remove the information — it just makes the discrimination harder to detect and audit, because you can no longer directly check whether the model's decisions correlate with the protected group; you would have to go hunting for every proxy separately.

A classic demonstration of proxy discrimination, unrelated to algorithms but essential background, is the 2004 field experiment by economists Marianne Bertrand and Sendhil Mullainathan, published in the American Economic Review. They sent out nearly identical fictitious resumes to real job postings in the United States, varying only the applicant's name — some resumes carried names perceived as distinctively White, others names perceived as distinctively African-American. The White-sounding names received roughly 50 percent more callbacks for interviews than the identical resumes with Black-sounding names. No "race" field existed anywhere on the resume. The name alone was a sufficient proxy. The exact same mechanism applies inside a machine learning model: if you delete the sensitive attribute but keep every correlated feature, the model can reconstruct the pattern anyway, often with startling accuracy, simply by combining several weak proxies into one strong signal.

Feedback Loops: When the Model's Own Decisions Poison Future Data

Bias does not have to be static. Consider a hypothetical, but realistic, credit-scoring system deployed by an Indian lender that uses pincode as one input feature, alongside income and repayment history, because pincode statistically correlates with default risk in the historical data. If certain pincodes were historically under-served by formal banking — meaning fewer people there had any credit history at all, simply because they had less access to credit in the first place, not because they were inherently less creditworthy — the model learns to rate applicants from those pincodes as higher risk. It then rejects more of them, or offers them worse terms. Those rejected applicants build no further credit history with this lender, so next year's training data has even less positive information from that pincode than it did before, and the model becomes even more confident in its rejection. The system is not just reflecting a historical inequality once; it is actively regenerating it, and each cycle makes the disparity look more "statistically justified" than the last, even though nothing about the underlying population changed — only the data the model was allowed to collect about them did. This is what researchers call a feedback loop, and it is one of the reasons a biased model can look increasingly accurate over time, on paper, while becoming more unfair in practice: the metric and the injustice are being generated by the same process.

What Can Actually Be Done

Fixes to algorithmic bias are usually grouped by where in the pipeline they intervene.

• Pre-processing: fix the training data before the model ever sees it — reweighting examples so under-represented groups count more during training, or resampling the dataset so groups appear in more balanced proportions.
• In-processing: change what the model optimizes for during training itself, by adding a fairness penalty term to the loss function alongside the accuracy term, so the optimizer is explicitly punished for producing unequal error rates, not just for being inaccurate.
• Post-processing: leave the trained model alone but adjust its decision thresholds separately per group after the fact. Moritz Hardt, Eric Price, and Nathan Srebro showed in 2016 that if you are willing to use different score cutoffs for different groups, you can mathematically guarantee equalized odds — at the direct, explicit cost of using group membership in the decision rule, which raises its own legal and ethical questions in contexts where treating groups differently by rule is itself prohibited.
• External audits: the Gender Shades study by Joy Buolamwini and Timnit Gebru (2018) tested commercial facial-analysis systems from Microsoft, IBM, and Face++, and found error rates for classifying gender from photos of darker-skinned women reaching as high as roughly 35 percent on some systems, versus well under 1 percent for lighter-skinned men on the same systems. That single independent audit, using a deliberately balanced test set the companies had not designed their systems around, led to public commitments from more than one of those companies to retrain and re-test their models on more representative data. Independent, adversarial measurement — testing a system on a dataset its builders did not curate — is often what actually forces a fix, more reliably than internal review.
• Regulation and disclosure requirements: the European Union's AI Act (2024) classifies systems used for credit scoring, recruitment, and law-enforcement risk assessment as "high-risk," legally requiring bias testing and documentation before deployment. In India, the Digital Personal Data Protection Act, 2023, and NITI Aayog's Responsible AI policy papers set out data-protection and fairness principles, though comprehensive binding rules specifically for algorithmic decision-making are still developing. Regulation cannot make the impossibility theorem go away — it can only force organizations to be explicit and accountable about which definition of fairness they chose, and why, instead of quietly picking whichever one made their system look best.

No single technique from this list "solves" fairness, because the impossibility result you derived above is a fact about probability, not a fact about any particular tool. Every real deployment is a documented, defensible choice about which error to tolerate and for whom — and making that choice consciously, instead of by accident, is the actual job.

CBSE and Competitive Exam Connections

CBSE's Artificial Intelligence curriculum treats AI Ethics as a core part of the "AI Project Cycle," and board-style questions on this unit are usually case-study based: you are given a scenario (a hiring tool, a loan model, a policing algorithm) and asked to identify the type of bias present and propose a mitigation, so being able to name historical, representation, measurement, and feedback-loop bias precisely, as in this chapter, is directly examinable. For competitive and Olympiad-style preparation (including newer AI-focused Olympiad tracks and GATE-foundation data science modules), the confusion-matrix algebra in this chapter — deriving PPV from base rate, FPR, and FNR using Bayes' rule — is the same machinery used in medical-test sensitivity and specificity problems and in signal-detection theory, so mastering the derivation here pays off well beyond ethics questions specifically.

Check Your Understanding

1. A hospital triage model has FPR = 10% and FNR = 15% for both men and women patients — equalized odds holds exactly. Male patients have a true base rate of illness of 40%; female patients have a true base rate of 20% (say, because the underlying condition is genuinely less common in women). Using the Bayes' rule formula from this chapter, is it possible for PPV to differ between the two groups even though FPR and FNR are equal? Explain why, in one sentence, without recomputing the exact numbers.
2. A company deletes the "gender" column from its resume-screening dataset and reports that its model is now "provably fair through unawareness." Using the concept of proxy variables, explain precisely why this claim is not something you should accept, and name one concrete feature that could still leak gender.
3. Take a group of 1000 loan applicants with a true default base rate of 25% (250 will genuinely default, 750 will not). A model applied to this group has FPR = 15% and FNR = 40%. Compute the number of true positives, false positives, true negatives, and false negatives, then compute PPV for this group. Show your working the way the chapter's worked example did.
4. Explain, in your own words, why ProPublica and Northpointe could both publish correct statistics about the same COMPAS dataset and still reach opposite conclusions about whether the tool was "fair." Name the two fairness definitions each side was implicitly using.
5. Give one original example (not from this chapter) of a feedback loop in an Indian context — a system whose own past decisions could plausibly bias the data it is trained on in the future — and explain the mechanism in two or three sentences.

Summary

Bias in a machine learning system is rarely a coding mistake; it is usually the model correctly learning a pattern from historical, representation, measurement, aggregation, or feedback-loop distortions already present in its training data, as the Amazon hiring tool demonstrated. A model's overall accuracy can hide sharply unequal error rates across groups, which is why fairness must be checked using group-wise confusion matrix metrics — FPR, FNR, TPR, and PPV — not a single headline number, as the COMPAS case showed. There are at least four legitimate, mutually reasonable mathematical definitions of fairness — demographic parity, equal opportunity, equalized odds, and predictive parity — and the Kleinberg-Mullainathan-Raghavan and Chouldechova impossibility results prove, via the Bayes' rule derivation of PPV from base rate and error rates, that you cannot generally satisfy calibration and equalized odds simultaneously when two groups have different true base rates. Deleting a sensitive attribute does not remove bias, because correlated proxy variables — pincode, name, school — can reconstruct the same signal, as both the Bertrand-Mullainathan resume study and real-world redlining-style patterns show. And because a deployed model's decisions can shape the very data used to retrain it, unaddressed bias tends to compound through feedback loops rather than stay fixed in place. The available fixes — pre-processing, in-processing, post-processing, independent audits, and regulation — each intervene at a different stage, but none of them make the underlying trade-off disappear; they only make the choice of which fairness definition to prioritize explicit, deliberate, and accountable instead of accidental.

Think About It

Think about this: How would you explain ai ethics and bias: the hard problems to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.