Before and beyond trust: reliance in medical AI

Abstract

Artificial intelligence (AI) is changing healthcare and the practice of medicine as data-driven science and machine-learning technologies, in particular, are contributing to a variety of medical and clinical tasks. Such advancements have also raised many questions, especially about public trust. As a response to these concerns there has been a concentrated effort from public bodies, policy-makers and technology companies leading the way in AI to address what is identified as a "public trust deficit". This paper argues that a focus on trust as the basis upon which a relationship between this new technology and the public is built is, at best, ineffective, at worst, inappropriate or even dangerous, as it diverts attention from what is actually needed to actively warrant trust. Instead of agonising about how to facilitate trust, a type of relationship which can leave those trusting vulnerable and exposed, we argue that efforts should be focused on the difficult and dynamic process of ensuring reliance underwritten by strong legal and regulatory frameworks. From there, trust could emerge but not merely as a means to an end. Instead, as something to work in practice towards; that is, the deserved result of an ongoing ethical relationship where there is the appropriate, enforceable and reliable regulatory infrastructure in place for problems, challenges and power asymmetries to be continuously accounted for and appropriately redressed.

The ethical challenges of artificial intelligence‐driven digital pathology

“…The ethics of AI-driven digital pathology Trustworthiness has been shown to be earned in a number of ways in medical contexts, for instance, through consent processes, professional integrity, and community engagement [35]. In the context of AI, it is argued that trustworthiness is earned by developing algorithms that are lawful, ethical, and robust and which respect principles of human autonomy, prevention of harm, fairness, and explicability (the last of which includes ideals of transparency and explainability) [45,46]. All of these norms will no doubt continue to be important to the public acceptability of AI.…”

“I don’t think people are ready to trust these algorithms at face value”: trust and the use of machine learning algorithms in the diagnosis of rare disease

“…guaranteed. Although Graham fails to clearly articulate the distinction between assured reliance, or confidence, and mere reliance [ 13 ], he does outline criteria for developing confidence-worthy systems for data sharing (here we substitute AI systems), arguing that they would involve: meaningful (i.e. understandable) transparency arrangements that can be checked, clear mechanisms of accountability, and assurances that the data involved are representative (i.e.…”

“…are working for the public good). Graham [ 30 ] argues that confidence, unlike trust, is not dependent upon A’s recognition of B’s good will [ 2 , 15 ], but, like reliance [ 13 ], is an enforceable obligation. Consequently, if AI development does not meet the conditions outlined above, for example, if it fails to meet the transparency requirement for confidence, then AI developers will be sanctioned.…”

“…It has been argued that while persons may rely on AI [ 13 ] they cannot enter into a trust relationship with it [ 14 ] as trust relationships are based on the recognition of another’s good will towards oneself and therefore, only human actors can be involved in trust relationships because “…one can only trust things that have wills” [ 15 : 14). At this point in the paper we do not wish to get into the argument about whether inanimate objects, such as AI tools, can participate in trust relationships (see [ 15 ]), nor indeed, whether non-human agents/actors (i.e.…”

Building Legal Datasets

“…Moreover, in practice g(H, y) is not the only metric to be optimized. Even assuming we care solely about economic value, profits, while correlated with (F1 score) performance, turn also on variables like user adoption and trust [28,9,15]. A perfectly accurate classifier that is never used generates no revenue.…”