Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance

This study addressed the cognitive impacts of providing correct and incorrect machine learning (ML) outputs in support of an object detection task. The study consisted of five experiments that manipulated the accuracy and importance of mock ML outputs. In each of the experiments, participants were given the T and L task with T-shaped targets and L-shaped distractors. They were tasked with categorizing each image as target present or target absent. In Experiment 1, they performed this task without the aid of ML outputs. In Experiments 2–5, they were shown images with bounding boxes, representing the output of an ML model. The outputs could be correct (hits and correct rejections), or they could be erroneous (false alarms and misses). Experiment 2 manipulated the overall accuracy of these mock ML outputs. Experiment 3 manipulated the proportion of different types of errors. Experiments 4 and 5 manipulated the importance of specific types of stimuli or model errors, as well as the framing of the task in terms of human or model performance. These experiments showed that model misses were consistently harder for participants to detect than model false alarms. In general, as the model’s performance increased, human performance increased as well, but in many cases the participants were more likely to overlook model errors when the model had high accuracy overall. Warning participants to be on the lookout for specific types of model errors had very little impact on their performance. Overall, our results emphasize the importance of considering human cognition when determining what level of model performance and types of model errors are acceptable for a given task. As machine learning (ML) algorithms are adopted into more and more contexts, it is important to consider how these tools impact their end users’ workflow and decision-making. In particular, it is crucial to consider how ML errors can impact human performance. Past research on automated systems shows that automation failures can be catastrophic. Yet much of the current research on ML research assumes that the algorithms will make very few errors. No algorithm will ever perform perfectly under all circumstances, but there has been relatively little research on the impact of ML errors on human decision-making. In high-consequence domains where ML stands to greatly benefit users, we must also address the potential impact of ML errors, especially if those errors are rare and difficult for users to notice. In this paper, we report a series of experiments that used tightly-controlled mock ML outputs to test the impact of ML errors on human performance in a target detection task. We found that models with low overall accuracy could improve human performance, but participants were less likely to notice ML errors when the errors were rare. Missed targets were harder for participants to detect than model false alarms, and higher proportions of misses among the ML errors led to lower human performance. Finally, warning participants to watch for missed targets did not improve their performance, although warning them about model errors in general had some benefit. Our findings have implications for future research and development of ML algorithms that can provide better support to human decision-making.