Disturbingly, researchers are unable to explain how it does that.

A study by researchers at the Massachusetts Institute of Technology and Harvard Medical School found that an AI model trained to read X-rays and CT scans could predict a person’s race with 90% accuracy. Scientists were surprised enough by this development to initially deem it a “mistake.” More so, because a doctor cannot tell a person’s race by looking at their X-ray.(Boston Globe)

An AI with unusual ability

This was first detected when scientists observed that an AI program for examining chest X-rays tended to miss signs of illness in Black patients.

Their curiosity piqued, the international team of scientists then proceeded to train an AI model on a set of X-rays and CT scans from various areas of the body, but with each image labelled for the person’s race. Notably, these images did not contain any other clues, such as skin color or type of hair, that hinted at the race of the person.

After training, the AI software was shown unlabelled images. Amazingly, the AI could now tell the person’s race with 90% accuracy, or more in some cases.

Marzyeh Ghassemi, an MIT assistant professor of electrical engineering and computer science, and coauthor of the paper on these findings, and her colleagues are unable to explain how the AI could accomplish this; however, they suspect the software was latching onto melanin, the pigment that determines the skin color of a person.

In people with darker skin, such as Blacks, the x-rays and CT scanners are perhaps able to detect higher melanin content and therefore establish some kind of correlation with race.

What is concerning is the likelihood of AI software being used in medicine that could have biases for race, something completely unintended by the creators of AI used on medical scans.

Related Story: GE Healthcare Launches AI-Powered, Cloud Imaging Solution For Radiologists