Bias, fairness and data quality

In short: AI models take over patterns from their data, including prejudices and gaps. If the data represents certain groups poorly, the model works worse for them.

In more detail: The model knows no morals, only statistics. If historical decisions discriminated (applications, loans), it learns that too.

In detail

• Sources: sampling bias (who was captured?), historical bias, measurement and label bias, feedback loops (recommendations reinforce themselves).
• Examples: face recognition with a higher error rate for dark skin, application filters that disadvantaged women, language models with stereotypes.
• Fairness notions: equal error rates per group, equal hit rates, equal treatment at equal qualification. They can contradict each other.
• Remedies: balanced data, evaluation separated by group, reweighting, tests (red teaming), documentation (datasheets, model cards).
• Humans in the loop: do not leave important decisions to the model alone.
• Law: the EU AI Act classifies high-risk systems (applications, credit, justice) strictly, see EU AI Act and rules for AI.

See also: Data, features and labels, Limits of AI: what models cannot do