The ethical problems in AI and ML research are not hypothetical. COMPAS deployed racially biased risk scores to judges. Facial recognition systems failed systematically on dark-skinned women. Cambridge Analytica harvested personal data from millions without their knowledge. Each case was a research and engineering decision that someone made — and that someone could have made differently.
Sign in to mark this article as read and track your progress.
Abstract ethical principles — fairness, transparency, non-maleficence — are easier to apply when you have seen what happens when they are violated. Each case below represents a real decision made by researchers, engineers, or product teams, with concrete consequences for real people. Reading them as case studies in research methodology means asking: what decisions were made at each stage of the project, what alternative decisions were available, and what ethical framework would have led to different outcomes?
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a commercial risk assessment tool used by courts in the United States to predict the likelihood that a defendant will re-offend before trial or after sentencing. Judges used COMPAS scores to inform bail and sentencing decisions.
In 2016, the investigative journalism organisation ProPublica published an analysis showing that COMPAS scores were racially biased: Black defendants were nearly twice as likely to be falsely flagged as high-risk compared to White defendants, while White defendants were more likely to be falsely flagged as low-risk (Angwin et al., 2016). The model was trained on historical criminal justice data — arrest records, conviction records — that already reflected decades of racially disparate policing and prosecution.
The research ethics lesson: using historical data as ground truth for a predictive model inherits the biases encoded in that history. When the outcome being predicted (re-offending) is measured by re-arrest, and arrest rates are not uniform across racial groups due to policing disparities, the model learns to predict who will be arrested rather than who will re-offend. Before deploying a predictive model in a high-stakes context, researchers and engineers must ask: what does this label actually measure, and whose values and biases are embedded in it?
A secondary lesson concerns transparency. Northpointe, the company behind COMPAS, refused to disclose the algorithm's features or weights, claiming trade secrecy. A defendant whose bail was set partly on the basis of a score from an algorithm they cannot examine cannot meaningfully contest it. This case established the principle that algorithmic transparency is not optional when a model is used to make decisions that affect a person's liberty.
Joy Buolamwini and Timnit Gebru published the Gender Shades study in 2018, auditing three commercial gender-classification systems (from Microsoft, IBM, and Face++) for accuracy across intersectional groups defined by gender (male/female) and skin tone (lighter/darker).
The findings were striking: all three systems performed substantially worse on darker-skinned women than on lighter-skinned men. Error rates for darker-skinned women were up to 34.7 percentage points higher than for lighter-skinned men in the worst case. The systems had been trained primarily on datasets containing disproportionately many lighter-skinned faces — a sampling bias that propagated directly into disparate real-world performance (Buolamwini and Gebru, 2018).
The research ethics lesson: the choice of training data is an ethical decision, not only a technical one. A dataset that over-represents one demographic group produces a model that works well for that group and poorly for underrepresented groups. The Gender Shades study introduced intersectional evaluation as a standard audit practice: rather than reporting a single aggregate accuracy figure, disaggregate performance across demographic subgroups, particularly groups that have historically been underrepresented in training data.
Buolamwini and Gebru's methodology — creating a balanced benchmark dataset (Pilot Parliaments Benchmark) and auditing commercial systems against it — is itself a model of research design. They could not access the training data of commercial systems; they tested outputs using a dataset they controlled. This demonstrates that meaningful external audit is possible even without access to proprietary systems.
In March 2016, Microsoft released Tay, a conversational AI chatbot deployed on Twitter. Within 16 hours, Tay was producing racist, sexist, and violent content. Microsoft shut it down. The system had been designed to learn from interactions with users, and coordinated groups of users had deliberately fed it offensive content to corrupt its outputs.
The research ethics lesson: AI systems deployed in adversarial environments — public-facing conversational systems, content moderation tools, any system where users have incentives to manipulate outputs — must be designed with adversarial robustness in mind before deployment. The failure to anticipate that a public chatbot could be targeted by coordinated manipulation was a foreseeable design failure, not an unforeseeable edge case.
The Tay case also illustrates the developer responsibility framework: the creator of a system that can cause harm retains ethical responsibility for foreseeable misuse, even when individual users are the proximate cause of the harm. This principle applies to security research tools (dual-use exploits), recommendation algorithms (that can be gamed to amplify misinformation), and any AI system deployed in an uncontrolled environment.
Between 2014 and 2018, the political consultancy Cambridge Analytica harvested personal data from approximately 87 million Facebook users without their informed consent. The mechanism was a third-party personality quiz app that, under Facebook's then-current API permissions, could collect data not only from the quiz-taker but from all of their friends — none of whom had consented to their data being collected.
Cambridge Analytica used this data to build psychographic profiles and target political advertising during the 2016 US presidential election and the UK Brexit referendum.
The research ethics lesson: this case is a systematic violation of the two most fundamental principles of data ethics:
Informed consent — the people whose data was used did not know it was being collected, and did not consent to its use for political micro-targeting. Data collected for one purpose (a personality quiz) was used for a completely different purpose (political profiling) without re-consenting the subjects.
Data minimisation — the platform's API allowed far more data to be collected than any legitimate research purpose required. Ethical data collection collects only what is necessary for the stated purpose.
The case led directly to the European Union's General Data Protection Regulation (GDPR) enforcement actions against Facebook, and is regularly cited in discussions of the DPDP Act 2023 in India (discussed in Unit 5's earlier article on informed consent and Indian law). For researchers, the lesson is that access to data does not constitute permission to use it. The fact that a platform's API allows certain queries does not mean those queries respect the autonomy of the people whose data is being queried.
ImageNet, the large-scale image dataset that drove the computer vision revolution after 2012, contains over 14 million labelled images. Later audits found that it included a "People" subtree with offensive category labels for people based on appearance, occupation, and perceived identity — categories scraped from WordNet without human review of their appropriateness (Yang et al., 2020).
The ImageNet team responded by removing the People subtree and developing procedures for ongoing dataset auditing. The case raised a question that now appears in every discussion of research data: who decides what is a valid category, and who is harmed by a category's existence?
A related development is the publication of Datasheets for Datasets (Gebru et al., 2018) — a proposal that every dataset released for research should be accompanied by a structured document describing its composition, collection process, intended uses, and known limitations, analogous to a data sheet for an electronic component. This framework is now a standard recommendation in ML research methodology.
Each of these cases resulted from specific decisions made at specific stages of a research or engineering project. The ethical obligations they illustrate map onto the research process:
| Stage | Obligation |
|---|---|
| Data collection | Obtain genuine informed consent; collect only what is necessary; document demographic composition |
| Dataset construction | Audit for demographic imbalance; document known limitations; publish a datasheet |
| Model evaluation | Disaggregate metrics across subgroups; test adversarial robustness; use appropriate baselines |
| Deployment | Anticipate foreseeable misuse; maintain oversight; define a decommissioning plan |
| Publication | Disclose limitations honestly; report negative results; do not overclaim generalisability |
Research ethics is not a checklist completed before a project starts. It is a continuous responsibility that runs from problem definition through post-deployment monitoring.
A research team trains a facial recognition system for attendance monitoring at a university and reports 96% accuracy. A student asks: "What is the accuracy for students who are not well-represented in the training data?" The team replies: "We did not track that." Identify the ethical failures in this research and design, describe what a responsible evaluation would have looked like, and explain which of the cases discussed in this article is most analogous to this situation and why.
