A framework to identify ethical concerns with ML-guided care workflows: a case study of mortality prediction to guide advance care planning

Abstract

Objective Identifying ethical concerns with ML applications to healthcare (ML-HCA) before problems arise is now a stated goal of ML design oversight groups and regulatory agencies. Lack of accepted standard methodology for ethical analysis, however, presents challenges. In this case study, we evaluate use of a stakeholder “values-collision” approach to identify consequential ethical challenges associated with an ML-HCA for advanced care planning (ACP). Identification of ethical challenges could guide revision and improvement of the ML-HCA. Materials and Methods We conducted semistructured interviews of the designers, clinician-users, affiliated administrators, and patients, and inductive qualitative analysis of transcribed interviews using modified grounded theory. Results Seventeen stakeholders were interviewed. Five “values-collisions”—where stakeholders disagreed about decisions with ethical implications—were identified: (1) end-of-life workflow and how model output is introduced; (2) which stakeholders receive predictions; (3) benefit-harm trade-offs; (4) whether the ML design team has a fiduciary relationship to patients and clinicians; and, (5) how and if to protect early deployment research from external pressures, like news scrutiny, before research is completed. Discussion From these findings, the ML design team prioritized: (1) alternative workflow implementation strategies; (2) clarification that prediction was only evaluated for ACP need, not other mortality-related ends; and (3) shielding research from scrutiny until endpoint driven studies were completed. Conclusion In this case study, our ethical analysis of this ML-HCA for ACP was able to identify multiple sites of intrastakeholder disagreement that mark areas of ethical and value tension. These findings provided a useful initial ethical screening.

President Biden’s Executive Order on Artificial Intelligence—Implications for Health Care Organizations

“…7 For uses greenlighted for deployment, follow-up assessments monitor how the concerns raised are addressed. The ethical review process includes quantitative assessments of model fairness for specific patient subgroups, qualitativeinterviewingwithmultiplestakeholders(AIdevelopers,clinical staff, hospital administrators, patient representatives) to surface ethical concerns, 8 and consultation with additional AI experts about recommendationsforaddressingtheseconcerns.DukeHealth'sABCDS (Algorithm-Based Clinical Decision Support) process requires proposers to develop and submit plans demonstrating the algorithm's clinical utility that include assessments of model fairness, 6 and work is underway to operationalize a wider range of ethical assessment criteria.…”

“…To be relevant, AI ethics assessments have to be fast, but gathering and reviewing the necessary information quickly requires substantial effort. 8 Finally, ethics assessments should incorporate patient and public perspectives, 9 yet the public has low familiarity with AI. Again, effective oversight will require investments in training participants.…”

“…A further challenge is the pressure many health care organizations feel to swiftly adopt AI tools to stay at the leading edge of innovation. To be relevant, AI ethics assessments have to be fast, but gathering and reviewing the necessary information quickly requires substantial effort . Finally, ethics assessments should incorporate patient and public perspectives, yet the public has low familiarity with AI.…”

Clearing the Fog: A Scoping Literature Review on the Ethical Issues Surrounding Artificial Intelligence-Based Medical Devices

“…The 73 included studies covered a broad spectrum of medical contexts, summarised with the main ethical issues raised and key findings in Supplementary Table S2. Briefly, the most frequently addressed areas were as follows: the general use of AI for healthcare (36 studies) , the use of AI in decision support systems (eight studies) [78][79][80][81][82][83][84][85][86][87][88][89][90][91][92][93][94], big data (four studies) [95][96][97][98], robotics (seven studies) [99][100][101][102][103][104][105] and adaptive AI (one study) [106]. The remaining studies addressed the following: rehabilitation [107], medical education [108], monitoring technology for the elderly [109], mental health [110], radiation technology [111], chatbots [112], health apps [113] and healthcare in low-and middleincome countries [114].…”

“…Cagliero et al performed a case study analysis and collected viewpoints from various stakeholders, noting that while designers appeared focused on the importance of algorithm validation, clinicians and patients had a desire to understand (at least at a certain level) how the AI system works [89]. This demonstrates a so-called 'values-collision', highlighting that it should not be taken for granted that all stakeholders assume the same viewpoint of what is required for transparency.…”