Intentional machines: A defence of trust in medical artificial intelligence

Abstract

Trust constitutes a fundamental strategy to deal with risks and uncertainty in complex societies. In line with the vast literature stressing the importance of trust in doctor–patient relationships, trust is therefore regularly suggested as a way of dealing with the risks of medical artificial intelligence (AI). Yet, this approach has come under charge from different angles. At least two lines of thought can be distinguished: (1) that trusting AI is conceptually confused, that is, that we cannot trust AI; and (2) that it is also dangerous, that is, that we should not trust AI—particularly if the stakes are as high as they routinely are in medicine. In this paper, we aim to defend a notion of trust in the context of medical AI against both charges. To do so, we highlight the technically mediated intentions manifest in AI systems, rendering trust a conceptually plausible stance for dealing with them. Based on literature from human–robot interactions, psychology and sociology, we then propose a novel model to analyse notions of trust, distinguishing between three aspects: reliability, competence, and intentions. We discuss each aspect and make suggestions regarding how medical AI may become worthy of our trust.

Misplaced Trust and Distrust: How Not to Engage with Medical Artificial Intelligence

“…The principles of the EU guidelines for trustworthy AI or the proposed criteria of computational reliabilism by Durán and Formanek can be seen as attempts to grasp trustworthiness. 36 Similarly, drawing on the Daniel Dennett's model of three stances, Starke et al 37 have recently suggested a dimensional model that can help disentangle different aspects of trustworthiness with view to reliability, competence, and intentions. In the context of medical AI, these aspects help scrutinize a system asking whether it works under the specified conditions, how well it performs in a particular task, and whether the "intentions" of the system, as imbued in it by the developers, are compatible with the intentions of the AI user.…”

“I don’t think people are ready to trust these algorithms at face value”: trust and the use of machine learning algorithms in the diagnosis of rare disease

“…Trust is a relational concept—a disposition or intentional attitude—which is associated with situations of uncertainty, relations of dependency and expectations about future behaviour/intentions [ 1 ]. If trust is regarded as a feature of relationships, such that A (truster) trusts B (trusted) to X [ 2 , 3 ], then trust is A’s attitude regarding B and trustworthiness is an aspect of B namely, “…the commitments, virtues, traits or features [of B] that ground justified or well-placed trust” [ 4 : 24]. Consequently, if B is perceived as, for example: dishonest, incapable of keeping a confidence and performing inconsistently and incompetently, then they will be deemed untrustworthy, as someone that one would not enter into a trust relationship with.…”

“…According to Bauer [ 5 ], trust is “..a belief formed as a result of probabilistic reasoning…a probability that quantifies a belief that the trusted person will in fact do what one is expecting her to do.” [p4], namely, A trusts B because she believes (there is a high probability) that B will X. Likewise, Starke et al [ 3 ] observe that trust is necessary to deal with uncertainty. They point out that in addition to the uncertainty about whether B will X or not X, trust comes into play when A occupies an uncertain or vulnerable relationship with B; A is dependent upon B’s good will to X [ 2 ].…”

“…Starke et al [ 3 ] argue that trust and trustworthiness can be seen as context dependent; whether A decides to place their trust in B will depend upon A’s willingness to trust, A’s perceptions of B, A’s prior experiences of B and similar situations and the wider context of the task, all of which will speak to B’s trustworthiness and A’s propensity to trust B in this moment. This suggests that the act of trusting is prone to fluctuation and is contingent and conditional.…”

“…In recent years it has been argued that AI tools, particularly those involving machine learning algorithms, are well placed to undertake some types of healthcare tasks, for example, they can reliably and accurately interpret images in pathology and radiology, thus freeing up clinicians’ time to engage in other aspects of care [ 10 ]. Many authors have suggested that as the use of AI becomes more pervasive, and opaque, black box computerised systems are used in clinical decision-making, the role of trust in, and the trustworthiness of, AI tools will need to be addressed [ 3 , 11 , 12 ].…”