Amazon's hiring AI learned to penalize the word "women's." A healthcare model recommended less care for Black patients because it used spending as a proxy for need. These weren't bugs — they were the model doing exactly what its data told it to. Responsible AI is the engineering discipline of catching that before it ships.
Learning Objectives
After this lesson, you will be able to:
Understand and compute three key fairness measures -- demographic parity, equalized odds, and calibration -- and know when each one applies
Spot the five stages where bias sneaks into ML systems and learn how to reduce it at each stage
Write a model card that honestly documents what a model can do, what it cannot do, and how fair it is
This is one of the most important lessons in the entire track -- not because of the math, but because of what is at stake. These are not hypothetical scenarios. Real people were harmed by biased AI systems. Learning this now means you can be part of the solution.
Before diving into the math of fairness, it is important to understand that these are not theoretical concerns. Real AI systems have caused real harm:
Amazon's Hiring AI (2018): Amazon built an AI recruiting tool trained on 10 years of resume data. Because the tech industry was historically male-dominated, the training data contained far more male resumes. The model learned that male-associated patterns (using words like "executed" or "captured," listing all-male sports) were positive signals. It systematically penalized resumes containing the word "women's" (as in "women's chess club captain") and downgraded graduates of two all-women's colleges. Amazon's engineers tried to fix the bias but could not eliminate it -- the model kept finding new proxy variables for gender. The project was scrapped entirely.
COMPAS Criminal Justice Algorithm: The COMPAS recidivism prediction tool, used by US courts to assess the risk of defendants reoffending, was found by ProPublica to exhibit significant racial bias. Black defendants were scored as higher risk even when they had identical criminal histories to white defendants. The algorithm's false positive rate (predicting someone would reoffend when they did not) was nearly twice as high for Black defendants (45%) as for white defendants (23%). Judges used these scores to make bail, sentencing, and parole decisions affecting thousands of lives.
Google Photos Labeling (2015): Google Photos' image classification system labeled photos of Black people as "gorillas." The root cause was a training dataset that was not diverse enough -- the model had been trained primarily on lighter-skinned faces and had insufficient representation to correctly classify darker-skinned faces. Google's initial fix was to remove the "gorilla" label entirely rather than solving the underlying bias problem, highlighting how difficult it is to retrofit fairness into a system that was not designed for it.
Every ML engineer should follow these five steps before deploying any model that affects people:
Step 1: Know Your Protected Groups.
Before writing a single line of code, identify which demographic groups could be affected by your model. For hiring: gender, race, age, disability status. For lending: race, national origin, marital status, age. For healthcare: race, gender, socioeconomic status. Consult legal counsel if unsure -- anti-discrimination laws vary by jurisdiction and application.
Step 2: Audit Your Training Data.
Compute demographic breakdowns of your training data. If 90% of your resume data is from male applicants, your model will learn male-biased patterns. If your medical imaging dataset is 85% from light-skinned patients, your model will be less accurate for darker-skinned patients. Document these distributions in a datasheet and decide whether to re-sample, re-weight, or collect additional data.
Step 3: Compute Fairness Metrics Before Launch.
Pick the fairness metric most appropriate for your use case (see the metrics section below) and compute it across all protected groups. Set a threshold (e.g., demographic parity ratio above 0.8) and treat a failure as a launch blocker, not a warning.
Step 4: Test for Intersectional Bias.
A model can be fair for women overall and fair for Black applicants overall but deeply unfair for Black women specifically. Always check subgroup intersections (gender x race, age x disability, etc.) -- this is where the most harmful biases hide.
Step 5: Monitor Continuously After Deployment.
Fairness is not a one-time check. User populations shift, data distributions change, and societal norms evolve. Set up automated fairness dashboards that compute metrics weekly and alert when any metric drops below your threshold. Schedule quarterly fairness reviews as a standing team practice.
A loan approval model rejects 40% of women vs 20% of men. Which fairness metric detects this?
The answer is B. Demographic parity directly measures whether approval (or rejection) rates are equal across groups. A 40% vs 20% rejection rate is a clear demographic parity violation. Equalized odds might or might not flag this, depending on the actual qualification rates. Calibration does not directly measure approval rates at all.
Definition: The approval rate should be the same across all demographic groups.
Formula: P(Y_hat = 1 | A = a) = P(Y_hat = 1 | A = b)
Where Y_hat is the model's prediction and A is the protected attribute (gender, race, age).
Intuition: If 60% of men are approved for loans, 60% of women should be too.
When to use: When you want equal representation in outcomes regardless of qualifications. Useful for advertising (everyone should see the same job ads) and resource allocation.
Limitation: Ignores qualifications entirely. If the base rate genuinely differs (e.g., more applicants from group A meet the criteria), enforcing demographic parity means either approving less-qualified people from group B or rejecting more-qualified people from group A.
Definition: The model should have equal true positive rates AND equal false positive rates across groups.
Formula
P(Y_hat = 1 | Y = 1, A = a) = P(Y_hat = 1 | Y = 1, A = b) (equal TPR)
P(Y_hat = 1 | Y = 0, A = a) = P(Y_hat = 1 | Y = 0, A = b) (equal FPR)
Intuition: Among people who actually deserve a loan, the approval rate should be equal across groups. Among people who do not deserve a loan, the rejection rate should also be equal.
When to use: When you want the model to be equally accurate for all groups. The gold standard for criminal justice, medical diagnosis, and other high-stakes decisions.
Limitation: Requires ground truth labels, which may themselves be biased. If historical labels reflect past discrimination, equalized odds preserves that discrimination.
Definition: When the model says "80% likely to repay," 80% of people in every demographic group should actually repay.
Formula: P(Y = 1 | S = s, A = a) = P(Y = 1 | S = s, A = b)
Where S is the model's confidence score.
Intuition: The model's confidence should mean the same thing regardless of who it is about.
When to use: When decision-makers use the model's confidence scores to make nuanced decisions (not just binary approve/reject). Important for risk scoring, medical probability estimates, and insurance pricing.
Limitation: A calibrated model can still have very different approval rates across groups if the underlying risk distributions differ.
In 2016, researchers proved that except in trivial cases, you cannot simultaneously satisfy demographic parity, equalized odds, and calibration. This is known as the impossibility theorem of fairness.
What this means in practice: You must choose which definition of fairness matters most for your specific application. There is no "universally fair" model.
Application
Recommended Metric
Why
Hiring
Equalized odds
Equal accuracy across groups matters most
Loan approval
Calibration + equalized odds
Scores should be meaningful AND accurate
Ad targeting
Demographic parity
Everyone should see the same opportunities
Medical diagnosis
Equalized odds
Missing a diagnosis should not depend on demographics
Criminal risk scoring
Calibration
Judges use scores to make decisions; scores must be equally reliable
Adjust the classification threshold for each group independently and watch how fairness metrics respond. Try to make all metrics pass simultaneously -- you will discover the impossibility theorem firsthand. Notice how Group B has systematically lower scores (simulating real-world bias), requiring a lower threshold to achieve equal true positive rates.
The problem: Your training data does not represent the population you are serving.
Examples
ImageNet contains mostly Western-centric images; models trained on it perform worse on images from Africa and Asia
Medical datasets historically underrepresent women and minorities; models trained on them are less accurate for these groups
Language models trained primarily on English text perform worse on African American Vernacular English
Mitigation: Audit data demographics before training. Actively collect data from underrepresented groups. Use stratified sampling to ensure balanced representation.
The problem: The features you choose encode societal biases.
Examples
ZIP code as a feature is a proxy for race in many US cities
Name-based features encode gender and ethnicity
"Years of experience" penalizes groups who were historically excluded from an industry
Mitigation: Audit features for proxy discrimination. Use causal analysis to identify which features are proxies for protected attributes. Remove or constrain proxy features.
The problem: Certain model architectures amplify existing biases in the data.
Examples
Word embeddings encode stereotypical associations ("doctor" is closer to "man," "nurse" is closer to "woman")
Language models generate more toxic text about certain demographic groups
Image generation models produce stereotypical representations of professions by gender and race
Mitigation: Use debiasing techniques during training (adversarial debiasing, constrained optimization). Fine-tune on balanced datasets. Apply post-processing corrections.
The problem: A model that is fair in the lab becomes unfair in the real world.
Examples
A facial recognition system tested in controlled lighting fails in real-world conditions that disproportionately affect darker skin tones
A chatbot trained on formal English performs worse for users who write informally, correlating with socioeconomic status
Feedback loops: a predictive policing model sends more police to certain neighborhoods, generating more arrests, which reinforces the model's predictions
Mitigation: Test in production conditions across demographic groups. Monitor fairness metrics continuously. Implement feedback loop detection and circuit breakers.
Re-sampling: Over-sample minority groups or under-sample majority groups to balance the training data.
Original dataset: 80% male, 20% female applicants
After oversampling: 50% male, 50% female (duplicate female examples)
After undersampling: 50% male, 50% female (remove male examples)
Try it! Create a tiny dataset yourself: 8 "male" applicants (6 hired, 2 rejected) and 2 "female" applicants (1 hired, 1 rejected). Compute the approval rate for each group. Is there a demographic parity gap? Now duplicate the female examples to create a balanced dataset and recompute. This hands-on exercise makes fairness metrics concrete.
Re-weighting: Assign higher importance to underrepresented groups during training without changing the dataset size.
Data augmentation: Generate synthetic examples for underrepresented groups using techniques like SMOTE or generative models.
Adversarial debiasing: Train a second model (adversary) to predict the protected attribute from the main model's predictions. Penalize the main model when the adversary succeeds. The main model learns representations that are useful for the task but uninformative about the protected attribute.
Constrained optimization: Add fairness constraints directly to the loss function. Instead of minimizing just prediction error, minimize prediction error subject to a fairness constraint (e.g., demographic parity ratio > 0.8).
Threshold adjustment: Use different classification thresholds for different groups to equalize error rates. If your model has a 10% false positive rate for men but 20% for women, lower the threshold for women to equalize.
Calibration: Re-calibrate confidence scores per group so that "80% likely" means the same thing for everyone.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
Tests · Compute approval rates for both groups. Verify demographic parity ratio is below 0.8. Check that equalized odds TPR difference exceeds 10%. Apply mitigation and verify improvement.
A model card is a short document (1-2 pages) that accompanies a trained model. It answers: who built this, what does it do, how well does it work, and for whom?
Required sections
Section
Content
Model Details
Architecture, training data size, version, date, authors
Intended Use
Primary use cases, users, and out-of-scope uses
Metrics
Performance metrics (accuracy, F1) broken down by demographic group
Training Data
Data sources, size, demographic composition, collection methodology
Evaluation Data
Separate evaluation set with demographic breakdown
Ethical Considerations
Known biases, risks, and mitigation steps taken
Caveats & Recommendations
Known limitations, deployment recommendations
Example model card entry (for a hiring classifier)
Model: ResumeRanker v2.3
Architecture: Fine-tuned BERT-base
Training data: 500K resumes from 2019-2023, balanced by gender
Intended use: Rank resumes for software engineering roles
Out-of-scope: Non-technical roles, executive hiring
Performance by group:
Overall accuracy: 87%
Male applicants: 88% accuracy, 82% TPR
Female applicants: 85% accuracy, 78% TPR
Non-binary applicants: 81% accuracy, 74% TPR
Known limitations:
- 4% TPR gap between male and female applicants
- Limited non-binary training data (2% of dataset)
- Trained on English resumes only
Mitigation: Threshold calibrated per group to equalize TPR within 2%
Responsible AI is no longer optional -- it is increasingly mandated by law.
Regulation
Region
Scope
Key Requirement
EU AI Act
EU
All AI systems
Risk classification, transparency, human oversight
NYC Local Law 144
NYC
Hiring AI
Annual bias audit by independent auditor
CCPA/CPRA
California
Consumer data
Right to opt out of automated decision-making
NIST AI RMF
US (voluntary)
All AI
Risk management framework, governance, mapping
Canada AIDA
The EU AI Act risk classification
Unacceptable risk (banned): Social scoring, real-time biometric surveillance, manipulative AI
High risk (heavy regulation): Hiring, credit scoring, criminal justice, medical devices, education assessment
Limited risk (transparency): Chatbots, deepfakes (must disclose AI involvement)
Minimal risk (no requirements): Spam filters, games, entertainment AI
For high-risk systems, the EU AI Act requires: conformity assessments, quality management systems, risk assessments, human oversight, technical documentation, and post-market monitoring. Non-compliance fines reach 35 million euros or 7% of global annual revenue.
Checking fairness once, at launch -- Fairness metrics change as your user population changes. A model that is fair at launch can drift into unfairness within months as usage patterns shift. Monitor continuously.
Optimizing for one fairness metric only -- Satisfying demographic parity can worsen equalized odds. Document which metric you prioritized and why. Acknowledge the trade-offs.
Ignoring intersectionality -- A model might be fair for women and fair for Black applicants but unfair for Black women specifically. Always check subgroup combinations, not just individual protected attributes.
Assuming debiasing is a one-time fix -- Bias mitigation is an ongoing process. New data, new features, new use cases, and changing societal norms all require re-evaluation.
Treating model cards as a checkbox -- A model card that says "no known biases" with no supporting data is worse than no model card. Document your actual findings, including the uncomfortable ones.
Fairness is plural, not singular. Demographic parity, equalized odds, and calibration are all valid definitions, but the impossibility theorem proves they cannot all be satisfied simultaneously; the right choice depends on the application context and its stakeholders
Bias enters at five stages. Data collection, labeling, feature engineering, model architecture, and deployment; fixing only one stage while ignoring the rest still produces a biased system
Mitigation spans the full pipeline. Pre-processing (re-sampling, re-weighting), in-processing (adversarial debiasing, fairness constraints), and post-processing (threshold adjustment) give you different levers for different stages
Model cards make accountability visible. A 1-2 page document with performance broken down by demographic group, intended use, limitations, and ethical considerations is the minimum bar for responsible deployment
Fairness is a monitoring problem, not a one-time audit. A fair-at-launch model drifts as the world changes; build fairness metrics into your production monitoring and re-audit after every significant model or data update
Fairness is a continuous engineering discipline, not a one-time audit. Next up: Security & Compliance -- the OWASP LLM Top 10, PII pipelines, and the threat models that protect what your fair model actually says in production.