Police AI May Worsen Bias Through Skewed Data, Researchers Warn
Researchers told BBC News that predictive policing AI trained on biased or flawed records may amplify discrimination, though others say modern systems can account for bias.
The story in focus
Researchers told BBC News that predictive policing AI trained on biased or flawed police records may amplify discrimination and create feedback loops, citing Sarah Brayne's Los Angeles research and an NYU study that found problematic police data available for training in nine of 13 US areas examined.
- Sarah Brayne's American Sociological Review paper found Los Angeles police increasingly used data to predict crime, with skewed inputs making predictions appear impartial
- An NYU study found police data from periods of unlawful or biased policing was available to train systems in nine of 13 US areas examined
- Dimitrios Kollias of Queen Mary University of London said modern AI can account for historical bias but still needs human control
- Experts called for independent checks of training data and greater human oversight before AI is used in policing
Feedback loops in crime data
Predictive crime mapping can become self-reinforcing, according to the BBC report. Crime figures are partly a record of where officers have looked, so patrolling shaded hotspots produces more recorded crime there, which then feeds the next prediction.
Sarah Brayne, a sociologist at the University of Texas at Austin, studied big-data use inside the Los Angeles Police Department. Her paper in the American Sociological Review found the force increasingly used data to predict crime rather than only respond to it, and that skewed inputs made the resulting predictions appear impartial. She described a loop in which models both predict crime and contribute to its future occurrence.
Flawed records and oversight
Researchers at New York University said many predictions may rely on dirty data, meaning police records shaped by corrupt, biased or unlawful practices. In their study, such data was available to train systems in nine of the 13 US areas examined; the researchers said the software cannot tell the data is flawed.
They argued that technical fixes from vendors are insufficient unless someone independently checks how the data was produced. Carolyn McKay of the University of Sydney Law School called for human oversight and scrutiny of state policing functions. Dimitrios Kollias of Queen Mary University of London said modern systems can combine learned information in new ways and account for historical bias, but that greater autonomy requires safeguards to keep behaviour safe and under meaningful human control.
What researchers propose
Suggested remedies include independent review of training data before deployment, removing biased records, and adding contextual data such as neglected areas with broken streetlights or missing services so agencies can consider environment rather than only patrol. McKay cautioned that place-based systems may still lead to over-policing of marginalised areas. Kollias said bias is not inevitable and that carefully designed, independently evaluated AI could make patterns more measurable.
How many US areas did the NYU study find had potentially flawed police data available for training?
Nine of the 13 areas examined. The researchers said records from periods of unlawful or biased policing were available to train the systems, which cannot detect that the data is flawed.
Sources & further reading
Help keep the record accurate. Send a correction.
