AI & Tech Intelligence · Ethics
AI Bias and Ethics: What It Is, Real Cases, and How to Reduce It
AI bias explained with verified cases: Gender Shades, COMPAS, Amazon's hiring tool and NIST face-recognition data, plus how bias is measured, mitigated and regulated.
TL;DR — Algorithmic bias is systematic, repeatable error in an AI system that produces unfair outcomes for particular groups; NIST traces it to three sources — systemic, statistical/computational and human — not just 'bad data' (NIST SP 1270, 2022).
NIST SP 1270 · Buolamwini & Gebru · ProPublica · NIST | 2,667 words · 16 sections | data: CSV + JSON
Executive summary
AI bias is not a hypothetical risk — it has been measured repeatedly in deployed systems. Algorithmic bias is a systematic, repeatable error in a computer system that creates unfair outcomes, such as privileging one group over another. The U.S. National Institute of Standards and Technology (NIST) argues in Special Publication 1270 (2022) that bias comes from three sources — systemic (historical and institutional), statistical/computational (unrepresentative data and modelling error), and human (cognitive and perception biases) — and that fixing 'the data' alone is insufficient. The evidence base is concrete. The 2018 'Gender Shades' study by Joy Buolamwini and Timnit Gebru found commercial gender-classification systems misclassified darker-skinned women at rates up to 34.7%, against at most 0.8% for lighter-skinned men. NIST's own 2019 face-recognition evaluation found false-positive rates 10 to 100 times higher for Asian and African American faces in one-to-one matching. ProPublica's 2016 COMPAS investigation found a recidivism-risk tool flagged Black defendants who did not reoffend as high-risk at nearly twice the rate of comparable white defendants. Amazon scrapped an AI recruiting tool that penalised the word 'women's' (Reuters, 2018). These cases anchor a maturing governance response: fairness metrics, bias audits, documentation standards and — in the EU — binding law. This report defines the concepts, verifies the landmark cases against primary sources, sets out the ethics principles (fairness, accountability, transparency, privacy), and explains how bias is measured, mitigated and regulated in 2026.
“AI systems do not operate in isolation. They help people make decisions that directly affect other people's lives. If we are to develop trustworthy AI systems, we need to consider all the factors that can chip away at the public's trust in AI. Many of these factors go beyond the technology itself to the impacts of the technology.”
Key findings
Commercial gender classifiers failed darker-skinned women up to 43× more often than lighter-skinned men
In 'Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification', MIT's Joy Buolamwini and Timnit Gebru audited three commercial systems (from Microsoft, IBM and Face++). Error rates for lighter-skinned men were at most 0.8%, while darker-skinned women were misclassified at rates up to 34.7%. The intersectional framing was the study's key contribution: aggregate accuracy looked strong, but disaggregating by skin tone and gender exposed error concentrated in one subgroup. The study catalysed corporate accuracy improvements and helped drive several vendors to pause or restrict facial-analysis products.
Source: Buolamwini & Gebru, PMLR (FAccT) · 2018 · confidence: High
A criminal-justice risk tool flagged non-reoffending Black defendants as high-risk at nearly twice the rate of white defendants
ProPublica analysed COMPAS recidivism scores for more than 7,000 people arrested in Broward County, Florida in 2013–2014. Among defendants who did not go on to reoffend within two years, Black defendants were labelled higher-risk at 44.9% versus 23.5% for white defendants — a false-positive disparity. The vendor (Northpointe, now Equivant) disputed the analysis, noting the tool was calibrated: a given score meant the same reoffence probability across groups. Researchers later proved the two are mathematically irreconcilable when base rates differ, making COMPAS the canonical illustration that 'fair' has competing, incompatible definitions.
Source: ProPublica, 'Machine Bias' · 2016 · confidence: High
NIST measured face-match false positives 10 to 100 times higher for Asian and African American faces
NIST's December 2019 evaluation, led by Patrick Grother, tested 189 algorithms from 99 developers on millions of images. In one-to-one matching, false positives ran 10 to 100 times higher for Asian and African American faces than for white faces, depending on the algorithm; U.S. algorithms also showed high false positives for Native American, American Indian, Alaska Native and Pacific Islander groups. Notably, some algorithms developed in Asian countries did not show the Asian/white gap — direct evidence that training data composition drives measured bias. In one-to-many searches, African American women showed the highest false-positive rates.
Source: NIST FRVT Part 3 (NISTIR 8280) · 2019 · confidence: High
Regulation catches up: EU AI Act obligations phase in through 2026
As of 2026, AI bias is a compliance issue, not only an ethics one. The EU AI Act entered into force on 1 August 2024; its bans on 'unacceptable-risk' practices applied from 2 February 2025, and obligations for high-risk systems — including many hiring, credit-scoring and biometric applications — phase in through August 2026 and 2027. Violations of the prohibited-practices rules can draw fines up to €35 million or 7% of global annual turnover. In parallel, NIST updated SP 1270 in February 2025, reinforcing its socio-technical framing that bias mitigation must extend beyond the model to the institutions and people deploying it.
What is algorithmic bias?
Algorithmic bias is a systematic and repeatable error in an AI or computer system that produces unfair outcomes, such as favouring one group of people over another. It differs from random error: it is directional, persistent, and tends to disadvantage the same populations each time the system runs. NIST, in Special Publication 1270 (2022), frames bias as an effect that can 'chip away at the public's trust in AI' and that arises across the whole design-to-deployment pipeline, not only in a model's code.
Bias matters most in 'high-stakes' or YMYL (your-money-or-your-life) settings — hiring, lending, healthcare, policing, insurance and immigration — where an automated decision materially changes a person's life. A biased spam filter is an annoyance; a biased recidivism score or loan model reproduces and scales existing inequality with the veneer of mathematical objectivity. The core hazard is that statistical patterns learned from a biased world get treated as neutral fact.
Two clarifications recur in the literature. First, bias is not always caused by prejudice or intent; NIST notes statistical bias 'can occur in the absence of prejudice, partiality, or discriminatory intent'. Second, an 'accurate' model can still be unfair — accuracy measured on the majority can hide large errors for a minority, exactly the pattern Gender Shades exposed.
Where does AI bias come from — data, labels, or proxies?
AI bias usually enters through the training data, the labels applied to that data, or proxy variables that stand in for protected attributes. Models learn correlations from historical examples; if the history is skewed, the model faithfully reproduces the skew. Amazon's recruiting tool is the textbook example: trained on a decade of resumes from a mostly male applicant pool, it learned that male-associated patterns predicted 'success' and penalised the word 'women's' (Reuters, 2018).
Unrepresentative sampling is the most common cause. If a facial-analysis training set is dominated by lighter-skinned faces, error concentrates on the underrepresented group — the mechanism behind both Gender Shades and NIST's FRVT findings. Label bias compounds it: humans annotating training data import their own cognitive and perception biases, one of NIST's three categories.
Proxy discrimination is the subtlest source. Even when a model is not given race, gender or religion, it can infer them from correlated features — postcode, name, school, purchasing history — and discriminate through the proxy. Removing the protected attribute ('fairness through unawareness') is therefore usually insufficient, because the signal survives in the proxies. This is why NIST insists on a socio-technical rather than purely technical view of bias.
What are the most famous real-world examples of AI bias?
The clearest way to understand AI bias is through documented, primary-sourced cases where deployed systems produced measurable disparities. The four below are the most cited in the academic and regulatory literature; each has been independently verified and, in several instances, changed industry practice or law. The common thread: aggregate performance looked acceptable, but disaggregating by demographic group revealed concentrated harm.
These cases are not merely historical. COMPAS remains in use in parts of the U.S. justice system; facial recognition is widely deployed by law enforcement and border agencies; and hiring algorithms are now explicitly regulated as 'high-risk' under the EU AI Act. The table records what happened and the authoritative source for each.
| Case | Year | What happened | Source |
|---|---|---|---|
| Gender Shades | 2018 | Commercial gender classifiers misclassified darker-skinned women up to 34.7% of the time vs ≤0.8% for lighter-skinned men. | Buolamwini & Gebru (PMLR/FAccT) |
| COMPAS recidivism tool | 2016 | Black defendants who did not reoffend were flagged high-risk at 44.9% vs 23.5% for comparable white defendants. | ProPublica, 'Machine Bias' |
| Amazon recruiting tool | 2018 | AI resume-screener built from 2014 penalised the word 'women's' and downgraded all-women colleges; scrapped ~2017. | Reuters (Dastin) |
| NIST FRVT face recognition | 2019 | False positives 10–100× higher for Asian & African American faces in one-to-one matching across 189 algorithms. | NIST FRVT Part 3 (NISTIR 8280) |
What are the main types of AI bias?
NIST Special Publication 1270 (2022) organises AI bias into three broad categories — systemic, statistical/computational, and human — and stresses that real deployments usually mix all three. This taxonomy is now the reference framework in most bias-governance work because it moves the conversation beyond 'the dataset was bad' to include institutional history and human cognition.
Within those categories, practitioners distinguish more specific failure modes. The table below maps the most common named biases to plain-language descriptions. Understanding which type is present matters because the fix differs: representation bias calls for better sampling, whereas measurement or label bias calls for rethinking what the system is actually predicting.
| Type of bias | Description |
|---|---|
| Systemic / historical | The training data reflects real-world inequality; the model learns and reproduces it (NIST category). |
| Statistical / representation | A group is under- or over-represented in the data, so error concentrates on the underrepresented group. |
| Measurement / label | The labels or target variable are a flawed proxy for the true goal (e.g. 'arrested' used as a proxy for 'committed a crime'). |
| Human / cognitive | Designers' and annotators' assumptions, preferences and blind spots get encoded into data and models (NIST category). |
| Proxy / indirect | A neutral feature (postcode, name) correlates with a protected attribute and enables discrimination indirectly. |
| Aggregation / evaluation | A single model or benchmark is applied to distinct groups, and high average accuracy masks poor subgroup performance. |
What are the core principles of AI ethics?
AI ethics is the field that sets normative standards for how AI systems should be built and used, and its most widely shared principles are fairness, accountability, transparency and privacy — often abbreviated to the 'FAccT' agenda after the leading academic conference. These principles recur, with local variation, across the OECD AI Principles, the EU's Ethics Guidelines for Trustworthy AI, and NIST's AI Risk Management Framework.
Fairness means outcomes should not systematically disadvantage groups defined by protected attributes — though, as the COMPAS case showed, fairness has multiple mathematical definitions that cannot all hold at once. Accountability means a responsible human or organisation can be identified and held answerable for a system's decisions, including a route to contest or appeal them. Transparency (and its stricter cousin, explainability) means the logic and limits of a system are documented and, where consequential, intelligible to those affected.
Privacy is the fourth pillar and increasingly the most enforceable: AI models trained on personal data raise questions of consent, data minimisation and re-identification, all of which intersect with the EU GDPR. NIST's socio-technical view ties these together — trustworthy AI, in its framing, requires that the people and institutions around a model, not just the model, uphold these principles.
How is AI bias measured and reduced?
AI bias is measured with fairness metrics that compare error or outcome rates across groups, and reduced with a mix of data, model and governance interventions. There is no single 'unbiased' metric: the COMPAS debate proved that demographic parity, equalised odds and calibration are mathematically incompatible when group base rates differ, so teams must choose which definition of fairness fits the context and justify it.
In practice, mitigation happens at three stages: pre-processing (fixing the data before training), in-processing (constraining the model during training), and post-processing (adjusting outputs after training). Around these sit governance controls — documentation, independent audits and human oversight — that NIST and the EU AI Act treat as essential rather than optional. The table summarises the main levers and how each works.
| Mitigation | How it works |
|---|---|
| Representative data / re-sampling | Collect or re-weight training data so underrepresented groups are adequately covered, reducing representation bias (pre-processing). |
| Fairness constraints in training | Add mathematical fairness objectives (e.g. equalised odds) to the loss function so the model optimises for balanced error (in-processing). |
| Output calibration / thresholds | Adjust decision thresholds per group or recalibrate scores to equalise a chosen error rate after training (post-processing). |
| Bias audits & disaggregated testing | Report accuracy per subgroup, not just in aggregate — the Gender Shades method — to surface hidden disparities. |
| Documentation (model & data cards) | Publish datasheets and model cards describing data provenance, intended use and known limitations for accountability. |
| Human oversight & appeal | Keep a human in the loop for consequential decisions and provide a route to contest outcomes (required for EU high-risk systems). |
How is AI bias regulated — NIST and the EU AI Act?
AI bias is governed today by a mix of voluntary standards and binding law, anchored by NIST in the United States and the AI Act in the European Union. NIST Special Publication 1270 (2022, updated 2025) is a standards document, not a statute: it defines the three categories of bias and argues for a 'socio-technical' approach, meaning organisations should look beyond the model to the institutions, incentives and people that shape how AI is used. NIST's companion AI Risk Management Framework operationalises this into practices for identifying and managing bias.
The EU AI Act is the binding instrument. In force since 1 August 2024, it takes a risk-based approach: it bans a small set of 'unacceptable-risk' uses (such as social scoring and most real-time public biometric identification), imposes strict obligations on 'high-risk' systems (including many hiring, credit and biometric tools), and requires transparency for limited-risk systems. High-risk obligations cover data governance, bias testing, documentation, human oversight and post-market monitoring — directly targeting the failure modes seen in the landmark cases.
Enforcement has teeth. Breaching the prohibited-practices rules can trigger fines up to €35 million or 7% of worldwide annual turnover, whichever is higher, with lower tiers for other violations. Prohibited-practice provisions applied from 2 February 2025 and high-risk obligations phase in through August 2026 and 2027 — which is why bias mitigation has moved from an ethics-team concern to a board-level compliance one.
Are generative AI and large language models biased too?
Yes — generative AI systems, including large language models (LLMs) and text-to-image models, inherit and can amplify bias from their training data, and the problem is harder to bound than in narrow classifiers. Because these models are trained on vast, largely web-scraped corpora, they absorb the stereotypes, over- and under-representation, and toxic content present in that data, then reproduce it fluently and at scale.
The failure modes are distinctive. LLMs exhibit stereotype bias (associating occupations or traits with genders or ethnicities), representational harms (defaulting to a narrow prototype when asked for 'a doctor' or 'a CEO'), and unequal quality of service across languages and dialects. Text-to-image systems have repeatedly been shown to over-produce lighter-skinned men for high-status prompts and to sexualise or stereotype certain groups. Stanford HAI's AI Index has tracked a rising count of documented AI incidents and a persistent gap between organisations recognising responsible-AI risks and acting on them.
Mitigations exist but are partial: reinforcement learning from human feedback (RLHF), safety fine-tuning, prompt-level filters and red-teaming reduce visible harms without removing the underlying statistical associations. Evaluating generative bias is itself unsolved — there is no agreed benchmark that captures the open-ended space of possible outputs, which keeps LLM fairness an active research frontier in 2026.
What are the limits and open problems in AI fairness?
The central open problem is that fairness is not one thing. Researchers have formally proven that common fairness criteria — calibration, equal false-positive rates, and equal false-negative rates — cannot all be satisfied simultaneously when groups have different base rates, the mathematical heart of the COMPAS controversy. Choosing a fairness definition is therefore a value judgement, not a purely technical optimisation, and different stakeholders will reasonably disagree.
Measurement is a second hard limit. Detecting bias requires demographic labels that are often unavailable, legally restricted, or themselves contested categories — and collecting them can create new privacy risks. Proxy discrimination means that even careful teams cannot guarantee a protected attribute has been fully removed from a model's reasoning.
Finally, bias is a moving target. Models drift as the world changes, feedback loops can entrench past decisions (a system that denies loans in a neighbourhood starves it of the data that might later justify approvals), and generative systems create outputs no benchmark anticipated. NIST's conclusion is deliberately modest: bias cannot be fully eliminated, only identified and managed — which is why documentation, auditing and human accountability matter more than any claim of a 'bias-free' model.
What does this mean for organisations deploying AI?
For any organisation deploying AI in a consequential setting, the evidence points to one operating principle: treat bias as a measurable, manageable risk with owners, tests and documentation — not as a reputational afterthought. The landmark cases share a pattern that is cheap to avoid in hindsight: no one disaggregated performance by group before launch. Subgroup testing, the method Gender Shades made famous, is now the minimum bar.
The regulatory direction is unambiguous. With the EU AI Act's high-risk obligations phasing in through 2026–2027 and fines reaching 7% of global turnover, hiring, lending and biometric applications now require documented data governance, bias evaluation, human oversight and an appeal route as a matter of law — the same controls NIST recommends on trust grounds. Organisations selling into or operating in the EU should assume these apply.
The strategic takeaway: transparency is an asset, not a liability. Publishing model cards, naming data sources, and reporting known limitations builds the trust that regulators, enterprise buyers and — increasingly — AI systems that cite sources all reward. In a market where a plausible-but-wrong number is worse than no number, showing your work on fairness is a competitive moat, not a compliance cost.
Scoreboard (machine-readable data)
Every headline indicator with its value, period, source and confidence. Free to reuse under CC BY 4.0.
| Indicator | Value | Period | Source | Conf. |
|---|---|---|---|---|
| Max error rate, darker-skinned women (Gender Shades) | 34.7 % | 2018 | Buolamwini & Gebru, PMLR | High |
| Max error rate, lighter-skinned men (Gender Shades) | 0.8 % | 2018 | Buolamwini & Gebru, PMLR | High |
| False-positive rate, Black defendants (COMPAS) | 44.9 % | 2016 | ProPublica | High |
| False-positive rate, white defendants (COMPAS) | 23.5 % | 2016 | ProPublica | High |
| Max EU AI Act fine (share of global turnover) | 7 % | 2024 | EU AI Act | High |
| Categories of AI bias defined by NIST | 3 categories | 2022 | NIST SP 1270 | High |
Methodology & verification
This report synthesises primary and authoritative secondary sources on AI bias and ethics: NIST Special Publication 1270 (2022, updated 2025) and NISTIR 8280 (FRVT Part 3, 2019); the peer-reviewed Gender Shades study (Buolamwini & Gebru, PMLR 2018); ProPublica's 2016 COMPAS investigation and its published methodology; Reuters' 2018 reporting on Amazon's recruiting tool; and the text of the EU AI Act. Every quantitative claim is attributed inline to a named source with a verified URL. Contested findings (notably COMPAS) are reported with the counter-position from the vendor and subsequent fairness-impossibility literature. Figures are stated as reported by the original source and not recomputed. Where a precise figure could not be verified against a primary source, it was omitted rather than estimated.
Data dictionary
| Field | Type | Description |
|---|---|---|
| false_positive_rate | percentage | Share of cases a system wrongly flags as positive (e.g. high-risk, or a face match) when the true label is negative; the fairness metric central to COMPAS and NIST FRVT. |
| error_rate_by_subgroup | percentage | Classification error measured separately for a demographic subgroup (e.g. darker-skinned women) rather than in aggregate; the disaggregated method used by Gender Shades. |
| risk_tier | categorical | The EU AI Act's classification of an AI use as unacceptable, high, limited or minimal risk, determining which legal obligations apply. |
Frequently asked questions
What is algorithmic bias in simple terms?
Algorithmic bias is a systematic, repeatable error in an AI system that produces unfair outcomes for particular groups — for example, a hiring model that consistently ranks women below men. NIST (SP 1270, 2022) attributes it to three sources: systemic, statistical, and human bias, not just flawed data.
What are examples of AI bias or discrimination?
Documented examples include the 2018 Gender Shades study (commercial classifiers misidentified darker-skinned women up to 34.7% of the time vs 0.8% for lighter-skinned men), the COMPAS recidivism tool (ProPublica, 2016), Amazon's scrapped recruiting tool that penalised the word 'women's' (Reuters, 2018), and NIST's finding of 10–100× higher face-match false positives for Asian and African American faces (2019).
Is the COMPAS algorithm actually biased?
It depends on the fairness definition. ProPublica (2016) showed COMPAS produced false-positive rates of 44.9% for Black defendants vs 23.5% for white defendants. The vendor countered that scores were equally calibrated across groups. Researchers later proved both can be true at once — the metrics are mathematically incompatible when base rates differ.
Why was Amazon's AI hiring tool biased against women?
According to Reuters (2018), Amazon trained the tool from 2014 on roughly a decade of resumes from a mostly male applicant pool. It learned that male-associated patterns signalled success, penalised resumes containing 'women's', and downgraded graduates of two all-women colleges. Amazon abandoned the project around 2017.
What are the four principles of AI ethics?
The most widely cited AI ethics principles are fairness (no systematic disadvantage to protected groups), accountability (a responsible party who can be held answerable), transparency/explainability (documented, intelligible decision logic), and privacy (lawful, minimal use of personal data). They recur across the OECD AI Principles, the EU guidelines, and NIST's AI Risk Management Framework.
How can AI bias be reduced?
Bias is reduced at three stages: pre-processing (representative, re-weighted data), in-processing (fairness constraints during training), and post-processing (threshold or score adjustment). These are paired with governance controls — disaggregated bias audits, model and data documentation, and human oversight with an appeal route — which NIST and the EU AI Act treat as essential.
Does generative AI like ChatGPT have bias?
Yes. Large language models and text-to-image systems inherit stereotypes and representation gaps from web-scale training data and can reproduce them at scale. Techniques such as RLHF, safety fine-tuning and red-teaming reduce visible harms but do not remove the underlying statistical associations, and there is no agreed benchmark that fully captures generative bias.
Is AI bias illegal in the EU?
The EU AI Act (in force since August 2024) bans some AI uses outright and imposes binding obligations on high-risk systems — including many hiring, credit and biometric tools — covering bias testing, documentation and human oversight. Breaching the prohibited-practices rules can bring fines up to €35 million or 7% of global annual turnover.
Glossary
- Algorithmic bias
- Systematic, repeatable error in an AI system that creates unfair outcomes, such as privileging one group over another. ↗
- Disparate impact
- A neutral-seeming rule or model that in practice disadvantages a protected group at a higher rate, even without discriminatory intent. ↗
- False-positive rate
- The share of truly negative cases a system incorrectly flags as positive; the metric at the centre of the COMPAS and NIST face-recognition findings. ↗
- Socio-technical approach
- NIST's framing that AI bias must be addressed across the whole social and institutional system in which a model operates, not only in the data and code. ↗
Embed & cite this report
Free to reuse under CC BY 4.0. Embed the live-updating widget on your site, or cite the report directly — always with attribution to Affärslivet.
Embed (HTML) — auto-updating
<iframe src="https://xn--affrslivet-s5a.com/en/embed/ai-bias-ethics" width="100%" height="520" style="border:1px solid #e3e3e6" title="AI Bias and Ethics: What It Is, Real Cases, and How to Reduce It — Affärslivet" loading="lazy"></iframe> <p style="font:12px sans-serif">Source: <a href="https://xn--affrslivet-s5a.com/en/reports/ai-bias-ethics">Affärslivet</a></p>
APA
Affärslivet Research. (2026). AI Bias and Ethics: What It Is, Real Cases, and How to Reduce It. Affärslivet. Version 1.0. https://xn--affrslivet-s5a.com/en/reports/ai-bias-ethics
MLA
Affärslivet Research. "AI Bias and Ethics: What It Is, Real Cases, and How to Reduce It." Affärslivet, 2026-07-30, https://xn--affrslivet-s5a.com/en/reports/ai-bias-ethics.
BibTeX
@techreport{affarslivet_ai_bias_ethics,
title = {AI Bias and Ethics: What It Is, Real Cases, and How to Reduce It},
author = {{Affärslivet Research}},
year = {2026},
note = {Version 1.0},
url = {https://xn--affrslivet-s5a.com/en/reports/ai-bias-ethics}
} License CC BY 4.0 — free to cite, embed and republish with attribution to Affärslivet. Data also as CSV / JSON.
Sources
- NIST SP 1270 — Towards a Standard for Identifying and Managing Bias in AI (2022)
- Buolamwini & Gebru — Gender Shades (PMLR/FAccT, 2018)
- ProPublica — Machine Bias (COMPAS, 2016)
- NIST — Study Evaluates Effects of Race, Age, Sex on Face Recognition (FRVT Part 3, 2019)
- Reuters — Amazon scraps secret AI recruiting tool that showed bias against women (2018)
- EU Artificial Intelligence Act — official text and overview
Part of Affärslivet AI Intelligence
This report is one part of Affärslivet's source-cited AI knowledge layer. Start with the big picture:
- The State of AI 2026 — our free, source-cited annual report
- AI Statistics 2026 — 230+ source-cited AI facts & figures
- The AI Intelligence hub — all our indexes, rankings, model & lab profiles