13 Dec When AI is Unfair: Understanding Bias
AI systems are deeply influenced by the data and design choices behind them, often carrying forward existing biases into their outcomes. Understanding fairness requires balancing metrics like statistical parity, equal opportunity, and predictive parity, each with its own trade-offs. To address these challenges, researchers apply mitigation strategies across the pipeline and examine real-world case studies such as COMPAS, hiring tools, and credit scoring.
We will discuss the following:
- Bias in, Bias out: Tracing bias through the AI pipeline
- Measuring Fairness: Statistical parity, equal opportunity, predictive parity, and their trade-offs
- Mitigation Strategies: Pre-processing, in-processing, and post-processing techniques
- Case Studies: Recidivism algorithms (COMPAS), hiring tools, credit scoring
1. Bias in, Bias out: Tracing bias through the AI pipeline
This topic explains how bias can enter and propagate at different stages of developing and deploying an AI system.
Data Bias
- Historical Bias: Real-world inequalities are captured in training data (e.g., hiring data showing gender disparities in certain roles).
- Representation Bias: Certain groups are underrepresented in the dataset.
- Measurement Bias: The way data is collected or labelled is flawed (e.g., using arrest records as a proxy for crime, which can reflect policing bias rather than actual offense rates).
- Aggregation Bias: Treating diverse groups as homogeneous, ignoring important subgroup differences.
Algorithmic Bias
- Learning Bias: The algorithm’s design or objective function may favor certain patterns (e.g., optimizing for overall accuracy can harm minority groups).
- Evaluation Bias: Testing the model on data that does not represent deployment conditions or using metrics that hide unfair outcomes.
User Interaction & Feedback Loop Bias
- Deployed systems influence user behavior, which generates new data (e.g., a hiring tool favoring certain profiles leads to more such candidates being hired, reinforcing the bias in future data).
- Automation Bias: Users over-trust algorithmic recommendations, reinforcing biased outcomes.
Bias is not just a “data problem”; it can arise from design choices, societal context, and interaction patterns. “Bias in, bias out” means that unfairness in any stage will likely manifest in the system’s outputs.
2. Measuring Fairness: Statistical parity, equal opportunity, predictive parity, and their trade-offs
This section covers formal mathematical definitions of fairness used to evaluate AI systems. Different definitions often conflict with each other and with accuracy.
Statistical Parity (Demographic Parity)
- A model’s predictions are equally distributed across protected groups (e.g., the same percentage of loan approvals for men and women).
- Critique: Ignores relevant differences between groups (e.g., qualifications). Can force equal outcomes even when base rates differ.
Equal Opportunity
- Among “qualified” individuals (true positives), all groups should have equal true positive rates (e.g., equally likely to be hired if they are qualified).
- Related: Equalized odds require both equal true positive rates and equal false positive rates across groups.
- More aligned with merit-based fairness, but requires defining “qualified” accurately.
Predictive Parity
- Among those predicted positively, the probability of being positive should be the same for all groups (e.g., if the algorithm predicts “low risk of recidivism,” the likelihood of not reoffending should be equal across races).
- Critique: Often incompatible with equal opportunity if base rates differ (known as Impossibility Theorem).
Trade-offs
- No single fairness metric is universally appropriate; choice depends on context and values.
- Fairness definitions often conflict with each other and with model accuracy (fairness-accuracy trade-off).
- Legal, ethical, and domain-specific considerations must guide which metric to prioritize.
3. Mitigation Strategies: Pre-processing, in-processing, and post-processing techniques
These are technical methods to reduce bias, categorized by when they are applied in the model development pipeline.
Pre-processing
- Modify the training data before model training to remove bias.
- Examples:
- Re-weighting samples to balance group representation.
- Changing feature values to decorrelate them from sensitive attributes (e.g., race, gender).
- Generating synthetic data for underrepresented groups.
- Advantage: Model-agnostic; works with any algorithm.
- Challenge: May distort real patterns or reduce data utility.
In-processing
- Modify the learning algorithm itself to incorporate fairness constraints or objectives.
- Examples:
- Adding fairness penalties to the loss function.
- Constraining optimization to satisfy statistical parity or equal opportunity.
- Using adversarial training where a second network tries to predict the sensitive attribute from the main model’s predictions, forcing the main model to learn features uncorrelated with the sensitive attribute.
- Advantage: Can directly optimize for fairness during training.
- Challenge: Often requires custom algorithms; can be computationally complex.
Post-processing
- Adjust model outputs after prediction to improve fairness.
- Examples:
- Applying different classification thresholds for different groups to achieve equal opportunity (threshold adjustment).
- Changing some positive predictions to negative (or vice versa) for specific groups to meet a fairness metric.
- Advantage: Easy to implement; no retraining needed; works with “black-box” models.
- Challenge: May break consistency between individuals; can reduce accuracy.
No technique is a silver bullet; the choice depends on the context, regulatory constraints, and available data.
4. Case Studies: Recidivism algorithms (COMPAS), hiring tools, credit scoring
These real-world examples illustrate how bias emerges, how fairness is measured (or ignored), and the societal impacts.
4.1 Recidivism Prediction (COMPAS)
- Issue: Used in US courts to predict a defendant’s likelihood of reoffending. ProPublica’s investigation (2016) found it was biased against Black defendants; a higher false positive rate (labelled as high risk but did not reoffend) compared to White defendants.
- Fairness Debate:
- Predictive parity: COMPAS supporters argued that accuracy (positive predictive value) was similar across races.
- Equal opportunity: Critics highlighted the disparity in false positive rates as unfair.
- Lesson: Shows the conflict between different fairness metrics. Also highlights how using arrest data can perpetuate policing biases.
4. 2 Hiring Tools
- Example: Amazon’s recruiting tool (discontinued) learned from historical hiring data and downgraded résumés containing the word “women’s” or graduates of women’s colleges.
- Issue: Automation of historical gender bias in tech hiring.
- Mitigation Attempts: Removing gender identifiers is a pre-processing step, but bias can linger via proxies (e.g., hobbies, certain experiences).
- Lesson: Even with sensitive attributes removed, models can infer them from other features (proxy discrimination). Requires careful auditing.
4.3 Credit Scoring
- Issue: Traditional and AI-driven credit scoring can disadvantage minority groups due to historical inequalities (e.g., less access to credit, lower average income).
- Proxy Bias: Using ZIP codes can act as a proxy for race due to residential segregation.
- Fair Lending Laws: In the US, regulations like the Equal Credit Opportunity Act (ECOA) prohibit discrimination. Lenders must show models do not create “disparate impact.”
- Mitigation: Some lenders use alternative data (e.g., rental payment history) to expand access, but these risks introducing new biases.
- Lesson: Balancing financial risk with fairness is a key challenge; transparency and regulatory compliance are critical.
Conclusion: This lesson moves from identifying sources of bias to measuring it formally, to mitigating it technically, and finally seeing it in practice. It emphasizes that fairness is a socio-technical challenge, requiring both technical tools and ethical, legal, and domain expertise to address responsibly.
If you liked the tutorial, spread the word and share the link and our website, Studyopedia, with others.
For Videos, Join Our YouTube Channel: Join Now
Read More:
- What is Deep Learning
- Feedforward Neural Networks (FNN)
- Convolutional Neural Network (CNN)
- Recurrent Neural Networks (RNN)
- Long short-term memory (LSTM)
- Generative Adversarial Networks (GANs)
No Comments