What are power shadows?

A working definition from MIT Sloan

power shadows (noun)

Biases or systemic exclusion of a soci­ety, as reflected in data sets.

As a graduate student working on facial analysis software, Joy Buolamwini, SM ’17, PhD ’22, encountered a problem: The software did not detect her dark skin, though it easily identified the faces of people with lighter skin. She had to wear a white mask to be detected by the computer.

The problem was the power shadows that are cast in datasets, Buolamwini, a computer scientist, writes in her book “Unmasking AI.” The dataset she used — created by a government agency with the intent of representing a diverse population — featured images of public figures, often elected officials. Around the world, white men have historically held political power, and that was reflected in the data. The software she used was trained on images in which people were predominantly white and male.

Power shadows have serious consequences. Machine learning models used to inform hiring decisions could reflect past biased decisions made by humans. Models used to diagnose medical conditions could be based on data from mostly white patients, resulting in poor outcomes for Black patients.

The first step to overcoming power shadows is being aware of them, according to Buolamwini. “We must also be intentional in our approach to developing technology that relies on data,” she writes.

Unmasking the bias in facial recognition algorithms

Working Definitions: Artificial Intelligence

MIT Sloan's Working Definitions explore the words and phrases behind emerging management ideas.

A lightbulb with the abbreviation "Ai" on it seems to be flying like a rocket ship

AI Executive Academy

In person at MIT Sloan

Register Now

Load More