When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making

Abstract

As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical to provide these decision makers with relevant inputs which can help them decide if and how to incorporate model predictions into their decision making. For instance, communicating the uncertainty associated with model predictions could potentially be helpful in this regard. However, there is little to no research that systematically explores if and how conveying predictive uncertainty impacts decision making. In this work, we carry out user studies to systematically assess how people respond to different types of predictive uncertainty i.e., posterior predictive distributions with different shapes and variances, in the context of ML assisted decision making. To the best of our knowledge, this work marks one of the first attempts at studying this question. Our results demonstrate that people are more likely to agree with a model prediction when they observe the corresponding uncertainty associated with the prediction. This finding holds regardless of the properties (shape or variance) of predictive uncertainty (posterior predictive distribution), suggesting that uncertainty is an effective tool for persuading humans to agree with model predictions. Furthermore, we also find that other factors such as domain expertise and familiarity with ML also play a role in determining how someone interprets and incorporates predictive uncertainty into their decision making.

Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies

“…Medicine & Healthcare Medical disease diagnosis [82], cancer image search [24], cancer image classification [73], COVID-19 diagnosis [137], balance disorder diagnosis [22], clinical notes annotation/medical coding [90], stroke rehabilitation assessment [86,87] Finance & Business Income prediction [62,115,144,155], loan approval [15,55,139], loan risk prediction [29], sales forecast [36,97], property price prediction [1,111], apartment price prediction [101] selecting overbooked airline passengers for re-routing [15], determining to freeze bank accounts due to money laundering suspicion [15], stock price prediction [16], marketing email prediction [60], dynamically pricing car insurance premiums [15].…”

“…On the other hand, regression tasks include forecasting monthly sales of Ahold Delhaize's stores [97], property price prediction [1,111], apartment rent prediction [101], and car insurance prediction [15]. Lastly, Biran and McKeown [16] asks participants to decide if they would buy a stock given various AI assistance.…”

“…Deep learning models Convolution Neural Networks [3,27], Recurrent Neural Networks [26,60], BERT [80], RoBERTa [13], VQA model (hybrid LSTM and CNN) [115], not specified [23,24,30,48,50,52,62,73,86,87,107,110,152] "Shallow" models Logistic regression [16,39,41,45,68,86,106,143], linear regression [28,37,101,111], generalized additive models * (GAMs) [1,13,39,43,49,128,135] decision trees/random forests [29,45,54,55,86,92,97,137,144,155], support-vector machines (SVMs)…”

“…In contrast to classification, uncertainty information of regression models is currently under-explored in human-AI decision making, with one recent exception [101] (though the effect of uncertainty information on decision making has long been studied outside the context of AI assistance, e.g. [70]).…”

“…Objective metrics of trust often focus on reliance as a direct outcome of trusting (i.e., how much people's decisions rely on or are influenced by the AI's), such as acceptance of model suggestions [13,16,22,26,35,80,81,90,94,96,101,143,152,153,155], likelihood to switch [58,96,101,109,152,155], weight of model advice [95,111], choice to use the Evaluation Subjective? Metrics Understanding Subjective Self-reported understanding [7,15,20,23,28,97,125,143,147], confidence in understanding [76], confidence in simulation [3,106], ease of understanding [60,111], intuitiveness [134], perceived transparency/interpretability [112,137]…”