A probabilistic theory of trust concerning artificial intelligence: can intelligent robots trust humans?

U-trustworthy Models. Reliability, Competence, and Confidence in Decision-Making

“…Calibration, which is closely related to the probability theory of trust (Afroogh 2023), is claimed to be as one of the key requirements for trustworthy AI and used by many practitioners (e.g., Safavi, Koutra, and Meij 2020;Tomani and Buettner 2021). By calibrating a classifier, it can be ensured that the predicted probabilities of the classifier more accurately reflect the true likelihood of each outcome, thereby increasing trust in the model's predictions.…”

“…To do so, we analyze the limitations of calibration and closely related metrics as a standalone condition for trustworthiness. This work reveals the shortcomings of the probabilistic framework (Afroogh 2023) in capturing the holistic notion of model trustworthiness. Additionally, our results uncover a limitation in the traditional metrics, such as accuracy, that are used for model comparison and hyperparameter tuning.…”

“…In the context of predictive models and decision-making, this notion translates into our reliance on a model that consistently demonstrates competence in achieving its goal in a decision-making task. In competence-based trust theories, trust is described through a three-part relationship "A trusts B to do X," (Horsburgh 1960;Ryan 2020;von Eschenbach 2021;Alvarado 2022;Afroogh 2023) where, in our case, A represents the end-user, B represents the predictive model, and X specifies the delegated task. What counts as good reasons for trusting is a matter of considerable debate, but at the very least, that B is capable or competent to do X is necessary for A to trust B.…”

“…Traditionally, probabilistic assessment or risk calibration has been considered a crucial aspect of model trustworthiness evaluation, as it ensures that predicted risks align with observed frequencies of outcomes and provides some information regarding the model uncertainty (Crowson, Atkinson, and Therneau 2016;Pleiss et al 2017;Hébert-Johnson et al 2018;La Cava, Lett, and Wan 2022;Afroogh 2023). However, its role in determining the trustworthiness of a model, as described above, remains an open question.…”

“…The Thirty-Eighth AAAI Conference on Artificial Intelligence prevailing belief that risk calibration alone is sufficient or necessary to establish trustworthiness (Barocas, Hardt, and Narayanan 2017;La Cava, Lett, and Wan 2022;Afroogh 2023). To do so, we analyze the limitations of calibration and closely related metrics as a standalone condition for trustworthiness.…”