Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways

Despite a proliferation of research on the ways that machine learning models can propagate harmful stereotypes, very little of this work is grounded in the psychological experiences of people exposed to such stereotypes. We use a case study of gender stereotypes in image search to examine how people react to machine learning errors. First, we use surveys to show that not all machine learning errors reflect stereotypes nor are equally harmful. Then, in experimental studies we randomly expose participants to stereotype-reinforcing, -violating, and -neutral machine learning errors. We find stereotype-reinforcing errors induce more experiential harm, while having minimal impact on participants’ cognitive beliefs, attitudes, or behaviors. This experiential harm impacts participants who are women more than those who are men. However, certain stereotype-violating errors are more experientially harmful for men, potentially due to perceived threats to masculinity. We conclude by proposing a more nuanced perspective on the harms of machine learning errors—one that depends on who is experiencing what harm and why.