Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare

Abstract

The need for AI systems to provide explanations for their behaviour is now widely recognised as key to their adoption. In this paper, we examine the problem of trustworthy AI and explore what delivering this means in practice, with a focus on healthcare applications. Work in this area typically treats trustworthy AI as a problem of Human-Computer Interaction involving the individual user and an AI system. However, we argue here that this overlooks the important part played by organisational accountability in how people reason about and trust AI in socio-technical settings. To illustrate the importance of organisational accountability, we present findings from ethnographic studies of breast cancer screening and cancer treatment planning in multidisciplinary team meetings to show how participants made themselves accountable both to each other and to the organisations of which they are members. We use these findings to enrich existing understandings of the requirements for trustworthy AI and to outline some candidate solutions to the problems of making AI accountable both to individual users and organisationally. We conclude by outlining the implications of this for future work on the development of trustworthy AI, including ways in which our proposed solutions may be re-used in different application settings.

The authors have requested that this preprint be removed from Research Square.

“…Our research on clinical diagnostic work is presented in the belief that the detailed understanding provided by an ethnographic approach and ethnomethodological analysis should be a precursor to the design and redesign and evaluation of clinical decision support systems. Diagnosis is entirely shaped by concerns about accountability (Procter et al, 2023); people are required to give explicit accounts or explanations for the different decisions that they make.…”

“…Accountability is embedded in the organisation of the process itself in the form of who has the right to speak in treatment planning meetings and the need to 'account' for particular or different and perhaps unusual evidence or interventions. Unusual results provoke debate whereby clinicians may request additional information from nurses, pathologists, radiologists, admin staff, or other clinicians (Procter et al, 2023). Accountability is afforded by and accomplished through the public visibility and sharing of documents, such as the patient record; CT scans, mammograms, x-rays, biopsy images, etc.…”

“…Questions that need to be addressed include: what design features of such tools might cause clinical professionals to 'trust' or 'mistrust' them? As with interpersonal trust, the need for evidence for the trustworthiness of technology is not achieved through a one-time act but must be an ongoing accomplishment (Procter et al, 2023). How can trust in these tools be sustained in and through interaction with them?…”

“…This reminds us how, as the technological innovations become entangled in the complexities of organisational working, so the challenges of for successful adoption correspondingly increase because of the requirement for their integration within existing and trusted work practices. Not least is the concern that while tools such as CDSSs may save diagnostic time and effort, this could be offset by the effort then required for continuing auditing of their trustworthiness (Procter et al, 2023).…”

“…The ethnomethodologically informed approach to ethnography we adopt is well described elsewhere ( (Clarke et al, 2001;Clarke et al, 2002;Clarke et al, 2003;Hartswood et al, 2002;2003a;2003b;2003c;Slack et al, 2007;Slack et al, 2010;Procter et al, 2022;Procter et al, 2023).…”