Ethics in AI: When Machines Make Unfair Decisions

A Hiring Algorithm That Learned to Punish the Word "Women's"

In 2014, engineers at Amazon began building an experimental tool to speed up hiring. The idea seemed sensible: feed the system about ten years of resumes the company had received, let it learn what a "successful" candidate's resume looked like, and then have it score new applicants automatically from one to five stars. By 2015, the team noticed something was badly wrong. The tool was quietly downgrading resumes that contained the word "women's" — as in "women's chess club captain" or "women's volleyball team, division 1." It also learned to rank graduates of two well-known all-women's colleges lower. Engineers tried patching the program to stop it from reacting to those exact words, but they could not guarantee it wouldn't find some other, subtler proxy for the same pattern. In 2018, Amazon quietly shut the project down. This is a real, well-documented case, first reported by Reuters in October 2018.

Nobody had written a line of code that said "prefer men." The problem was the training data. Because the technology industry had hired far more men than women over the previous decade, most of the resumes on file at Amazon — and especially most of the resumes that had eventually led to a job offer — belonged to men. The algorithm did exactly the job it was built to do: search thousands of past resumes for whatever pattern best predicted "this person got hired." It found one. That pattern happened to correlate strongly with gender, so the model, in effect, learned that being male was a point in an applicant's favour.

This is the central idea of this chapter, and it is worth stating precisely before we go any further: an AI system does not usually become unfair because a programmer is prejudiced. It becomes unfair because it is a pattern-finding machine, and the patterns sitting inside real-world data already carry the fingerprints of human history — including human unfairness. Learning exactly how that happens, how to detect it with arithmetic, and how to reduce it, is a genuine computer science skill, not just a moral lecture bolted onto a CS chapter.

From If-Else Rules to Learned Scores: What Is an AI "Decision," Really?

Before phones had face-unlock, a security guard decided who could enter a building by checking an ID card against a printed list. That is a decision made entirely by a human, following a rule the human also wrote and can explain in one sentence: "let them in if their name is on this list."

The next step up is a rule-based program. A simple example: a canteen app that decides whether a student gets a free lunch coupon. A programmer writes:

if family_income < 96000:      # rupees per year
    give_coupon = True
else:
    give_coupon = False

This is still fully a human decision — a human chose the number 96000 and the rule "less than." The computer just applies it fast. Every input and output can be explained in one line.

Most modern AI systems used for hiring, loans, admissions, or bail decisions work differently. Instead of a human writing the rule directly, the system is shown thousands of past examples — called training data — each with some features (income, age, marks, pin code, years of work experience) and a known label (was this loan repaid? was this employee rated "high performer" after one year? was this defendant re-arrested?). An algorithm searches for a mathematical rule that best matches the labels using the features, usually by giving each feature a weight (a multiplier showing how much it should count) and adding the weighted features into a score. If the score crosses a threshold, the decision flips from reject to approve. This process of searching for good weights from examples is called training a model. Once trained, the model applies its learned weights to brand-new people it has never seen.

Here is the crucial difference from the canteen rule: nobody sat down and chose the weights by hand. The weights emerged from whatever pattern best matched the historical labels — and if those historical labels were produced by decades of unequal human decisions, the "best matching" weights will faithfully reproduce that inequality. The algorithm is not lying, and it is not malfunctioning. It is doing statistics correctly on a record of an unfair past.

A Worked Example: Building a Toy Loan-Approval Score

Let's make this concrete with numbers you can check by hand. Imagine a tiny (entirely hypothetical, built for this lesson) bank branch with two localities in its service area, Area X and Area Y. For years, human loan officers approved loans in Area X far more freely than in Area Y, even for applicants with the same income and the same employment history — an old, informal bias against the community concentrated in Area Y. This pattern of unequal credit access along community lines is not invented out of thin air as a concept: Indian economists have documented similar patterns of discrimination in access to jobs along caste and religious lines (for example, Sukhadeo Thorat and Paul Attewell's 2007 study, published in the Economic and Political Weekly, sent matched fictitious job applications differing only in the caste- or religion-signalling name on the resume, and found applicants with Dalit or Muslim names received noticeably fewer callbacks than equally-qualified applicants with upper-caste Hindu names). Our numbers below are a simplified, made-up dataset built only to let you compute the effect yourself — not a real bank's data.

Here are eight past applicants. Income is in thousands of rupees per month; "years employed" means years in a stable job. Notice that A1–A4 (Area X) have exactly the same income and years-employed as A5–A8 (Area Y), pair for pair:

ID   Area   Income('000s)  Years employed  Historical decision
A1    X          30              4          Approved
A2    X          25              2          Approved
A3    X          20              1          Rejected
A4    X          15              3          Rejected
A5    Y          30              4          Rejected
A6    Y          25              2          Rejected
A7    Y          20              1          Rejected
A8    Y          15              3          Rejected

Now suppose a bank builds a simple scoring algorithm and trains it on this history. Because the historical labels favour Area X so strongly, the training process assigns Area X a bonus weight of 15 points — it has genuinely found that "Area X" is a strong predictor of "Approved" in the data it was shown, and mathematically, it is right to use it. The resulting model, in plain code, looks like this:

def predict_approval(income, years_employed, pincode):
    risk_score = income + (years_employed * 2)
    if 400001 <= pincode <= 400050:      # Area X pin codes
        risk_score += 15                  # learned "area bonus"
    return "Approve" if risk_score >= 40 else "Reject"

# A1 and A5 have IDENTICAL income and years_employed
print(predict_approval(30, 4, 400010))   # Area X pin code
print(predict_approval(30, 4, 400060))   # Area Y pin code

Let's trace this line by line for the two calls. For A1: risk_score = 30 + (4*2) = 38; the pincode 400010 falls in the 400001–400050 range, so risk_score += 15 makes it 53; since 53 >= 40, the function returns "Approve". For A5: risk_score = 30 + (4*2) = 38 — identical so far, because income and years employed are identical — but pincode 400060 does not fall in the Area X range, so no bonus is added; risk_score stays 38, and since 38 >= 40 is False, the function returns "Reject". Two applicants with the exact same financial profile get opposite outcomes, purely because of a five-digit pin code.

Common Misconception: "Just Delete the Sensitive Column"

A very natural first reaction is: "The fix is easy — just don't let the algorithm see 'area' or 'community' at all." This idea has a name, fairness through unawareness, and it is one of the most common mistakes in applied AI, so it is worth correcting explicitly. The pin code in our example is a proxy variable: a feature that is not itself the protected attribute (community, gender, religion) but is so strongly correlated with it that using the proxy has almost the same effect as using the protected attribute directly. In many Indian cities, pin code, surname, school attended, or even the specific vernacular used on a resume can each act as a proxy for caste, religion, or region, because Indian neighbourhoods, schools, and social networks are often still segregated along those very lines. Even if our loan algorithm were forbidden from ever reading a "community" field, it could reconstruct almost the same discriminatory pattern using pin code alone — which is exactly what happened in our worked example above. Removing one column does not remove the underlying pattern the algorithm is searching for; it only removes one convenient handle for finding it, and a large enough dataset usually offers several other handles.

A second common misconception, closely related, is that "math is neutral, so an algorithm can't be prejudiced the way a person can." Addition and multiplication are indeed neutral operations. But the inputs to that arithmetic — the training data and the labels — are records of human decisions, and human decisions have never been perfectly fair. Feeding biased history into a neutral formula produces a formula that is very good at reproducing that bias, consistently, at scale, and without anyone in the loop feeling responsible for any single decision. That combination — consistency, scale, and diffuse responsibility — is precisely why algorithmic bias deserves careful study rather than a shrug.

Measuring Unfairness: Why "Overall Accuracy" Can Hide the Real Problem

To catch unfairness like the one above, we need a way to measure it that doesn't rely on just eyeballing the code. Computer scientists do this by defining a ground truth — an independent, non-proxy measure of what the "correct" decision should have been — and then comparing the model's decisions to that ground truth, separately for each group.

In our toy example, let's define ground truth as: an applicant is genuinely creditworthy if income + years_employed*2 >= 25 — a rule that deliberately does not look at area at all. Applying this to our eight applicants: A1 (38), A2 (29), A5 (38), and A6 (29) are all genuinely creditworthy (≥ 25); A3 (22), A4 (21), A7 (22), and A8 (21) are not.

Now compare this ground truth to what our biased model actually decided. A1 and A2 (Area X) were creditworthy and were correctly approved. A5 and A6 (Area Y) were equally creditworthy — identical scores to A1 and A2 — but were incorrectly rejected. In the language of statistics, a "creditworthy applicant who gets rejected" is called a false negative. The false negative rate for a group is the fraction of that group's genuinely-qualified members who were wrongly turned away.

For Area X: 2 applicants were genuinely creditworthy (A1, A2), and 0 of them were wrongly rejected, so the false negative rate is 0 out of 2, which is 0%. For Area Y: 2 applicants were genuinely creditworthy (A5, A6), and both were wrongly rejected, so the false negative rate is 2 out of 2, which is 100%.

Here is the part that makes this worth teaching carefully: if you only looked at overall accuracy, the model would look almost fine. Out of all 8 applicants, 6 decisions matched the ground truth (A1, A2, A3, A4, A7, A8 were all decided correctly), giving an overall accuracy of 6/8 = 75%. A busy manager glancing at a single "75% accurate" dashboard number might approve this model for deployment. But that single number completely hides the fact that every one of the model's mistakes landed on Area Y, and that a genuinely qualified Area Y applicant had effectively no chance of approval. This is exactly the pattern journalists at ProPublica reported in 2016 when they investigated COMPAS, a risk-assessment algorithm then used by some US courts to help judges decide bail and sentencing: the tool's overall accuracy looked similar across racial groups, but Black defendants who did not go on to re-offend were far more likely to be wrongly flagged as "high risk" than white defendants who did not re-offend, while white defendants who did re-offend were more likely to be wrongly flagged as "low risk." Equal overall accuracy does not mean equal treatment — you have to check the error rates within each group.

Feedback Loops: When a Biased Decision Creates the Evidence for the Next One

Bias can also get worse over time on its own, through what computer scientists call a feedback loop. Consider a predictive-policing tool: an algorithm is fed historical crime-report data and outputs which neighbourhoods should receive extra police patrols. If, historically, a particular neighbourhood was already over-patrolled — perhaps for reasons unrelated to actual crime rates, such as older biased policies — then more officers were present there, which meant more minor offences were noticed and recorded there, simply because more people were watching. The algorithm, trained on this recorded data, concludes that neighbourhood is "high risk" and recommends sending even more patrols there. More patrols record even more incidents. The neighbourhood's data-recorded crime rate climbs, not because real underlying behaviour changed, but because the measurement process itself became self-reinforcing. Meanwhile a neighbourhood that was under-patrolled continues to look artificially "low risk," even if real crime there is just as common but rarely observed.

The same loop structure appears outside policing. A hiring algorithm that rejects certain candidates never finds out whether those candidates would actually have been excellent employees — it only ever gets to "see" the outcomes of people it approved, so its own past mistakes are invisible to it and never get corrected by new data. This is why a biased AI system, left running without human oversight, usually does not average out to fairness over time; it tends to lock in and amplify whatever pattern it started with.

Real Cases Every Student Should Recognise

The Amazon hiring tool and the COMPAS court-risk tool are two documented cases; here are a few more that show the same handful of underlying mechanisms — biased training data, proxy variables, and feedback loops — showing up in very different domains.

• Facial recognition, "Gender Shades" (2018): researchers Joy Buolamwini and Timnit Gebru tested three commercial face-analysis systems and found their error rate for classifying the gender of light-skinned men was under 1%, while their error rate for classifying darker-skinned women was over 20%. The likely cause: the photo datasets used to train these systems contained far more images of light-skinned faces than dark-skinned faces, so the models simply had far less practice on the harder, under-represented case.
• The UK's 2020 exam-grading algorithm: when COVID-19 cancelled England's A-level exams, the exam regulator Ofqual used a statistical formula to convert teachers' predicted grades into final grades, giving significant weight to each school's historical grade distribution over the previous three years. Students at schools with weaker past results — disproportionately larger state schools — had their teacher-predicted grades pulled down more often than students at schools with small class sizes and strong past results, which were often private schools. Public protest led the regulator to scrap the algorithm within days and revert to teachers' original predicted grades.
• Biometric authentication in Indian welfare delivery: several field studies of Aadhaar-linked fingerprint verification for the Public Distribution System and employment-guarantee wage payments found that manual labourers and elderly people, whose fingerprints are often worn down by decades of physical work, faced higher authentication-failure rates than office workers — sometimes resulting in denied or delayed rations and wages, until exception-handling procedures were added. Here the "unfairness" was not a discriminatory score but a hardware and training-data limitation: fingerprint-matching systems perform worse on precisely the population most dependent on the welfare scheme.

Notice the pattern across all four cases (Amazon, COMPAS, Gender Shades, Ofqual): none of them required a programmer to write a biased rule on purpose. Every one of them came from training data, feature choices, or measurement methods that quietly under-represented or mismeasured a particular group.

How Do Computer Scientists Actually Reduce This?

There is no single line of code that "makes an AI fair," because fairness is not one fixed number — researchers have proposed many mathematical definitions of fairness (equal accuracy, equal false-positive rates, equal false-negative rates, equal approval rates) and it is mathematically proven that a model generally cannot satisfy all of them simultaneously unless the underlying groups are identical in every relevant way. Still, real engineering teams reduce harm using a combination of concrete practices:

• Group-wise auditing before deployment: compute metrics like the false-negative rate example above, separately for every relevant group, before a model ever makes a real decision — exactly the arithmetic you just did by hand.
• Checking for proxy variables: testing whether any "neutral-looking" feature (pin code, school, surname, first name) is strongly correlated with a protected attribute, and either removing it or specifically correcting for its effect.
• Representative data collection: deliberately gathering more examples from under-represented groups instead of just using whatever historical data happens to already exist, which is exactly the fix the Gender Shades researchers recommended after their study.
• Keeping a human in the loop: using the model's score as one input to a human decision-maker who can override it, rather than letting the score alone make the final call, especially for high-stakes decisions like bail, hiring, or loan approval.
• Transparency and the right to an explanation: being able to tell an affected person which features drove their specific score, so mistakes and proxy effects can be challenged and corrected.

India's own policy institute, NITI Aayog, published its "Responsible AI for All" strategy documents in 2021, listing principles including equality, non-discrimination, reliability, and accountability that Indian AI systems — including ones used by government and by Indian companies — are expected to work towards. As a CBSE Computer Science or Informatics Practices student, this is exactly the kind of ethical-reasoning-plus-arithmetic question you should expect: not "is AI good or bad," but "given this data and this rule, compute the error rate for each group, and explain where the unfairness entered the pipeline."

Check Your Understanding

1. In the loan example, suppose a ninth applicant, A9, is in Area Y with income 25 and years_employed 2. Using the code in this chapter, compute A9's risk_score and state whether the function returns "Approve" or "Reject." Show your addition.
2. Explain in your own words why deleting the "area" column from the training data would not, by itself, fix the unfairness you found in Question 1's scenario. Use the word "proxy" in your answer.
3. A model has 90% overall accuracy on Group P and 90% overall accuracy on Group Q. A classmate claims this proves the model treats both groups fairly. Using the false-negative-rate idea from this chapter, explain why the classmate could still be wrong, and describe what additional numbers you would ask to see.
4. Give one original example (not from this chapter) of a feature that could act as a proxy variable for religion, caste, or region in an Indian context, and explain why.
5. Multiple choice: A predictive-policing algorithm sends more patrols to a neighbourhood because it was recorded as "high crime" in past data, more patrols record more minor offences there, and the algorithm sends even more patrols next cycle. This chain of events is best described as: (a) fairness through unawareness (b) a feedback loop (c) a rule-based system (d) a ground-truth label. Justify your choice in one sentence.

Summary

An AI decision, whether it approves a loan, ranks a resume, or flags a court defendant as "high risk," is usually the output of a model that learned weights from historical examples rather than a rule a human wrote by hand. When that history reflects real-world inequality, the learned weights reproduce it faithfully and consistently, even without any programmer intending unfairness. Simply deleting a sensitive column does not solve this, because other, seemingly neutral features — pin code, school, surname — can act as proxy variables that let the same pattern back in. Overall accuracy can also hide unfairness: a model can look highly accurate on average while its errors are concentrated almost entirely on one group, which is why computer scientists compute metrics like the false-negative rate separately for each group rather than trusting a single combined number. Left unchecked, biased decisions can also feed back into future training data, creating a loop that reinforces itself over time. Reducing this harm is an active, ongoing engineering discipline — auditing group-wise error rates, hunting for proxy variables, collecting more representative data, keeping humans in the loop for high-stakes calls, and demanding explanations for individual decisions — not a switch you can flip once and forget.