Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations

Abstract

The lack of transparency is one of the artificial intelligence (AI)'s fundamental challenges, but the concept of transparency might be even more opaque than AI itself. Researchers in different fields who attempt to provide the solutions to improve AI's transparency articulate different but neighboring concepts that include, besides transparency, explainability and interpretability. Yet, there is no common taxonomy neither within one field (such as data science) nor between different fields (law and data science). In certain areas like healthcare, the requirements of transparency are crucial since the decisions directly affect people's lives. In this paper, we suggest an interdisciplinary vision on how to tackle the issue of AI's transparency in healthcare, and we propose a single point of reference for both legal scholars and data scientists on transparency and related concepts. Based on the analysis of the European Union (EU) legislation and literature in computer science, we submit that transparency shall be considered the “way of thinking” and umbrella concept characterizing the process of AI's development and use. Transparency shall be achieved through a set of measures such as interpretability and explainability, communication, auditability, traceability, information provision, record-keeping, data governance and management, and documentation. This approach to deal with transparency is of general nature, but transparency measures shall be always contextualized. By analyzing transparency in the healthcare context, we submit that it shall be viewed as a system of accountabilities of involved subjects (AI developers, healthcare professionals, and patients) distributed at different layers (insider, internal, and external layers, respectively). The transparency-related accountabilities shall be built-in into the existing accountability picture which justifies the need to investigate the relevant legal frameworks. These frameworks correspond to different layers of the transparency system. The requirement of informed medical consent correlates to the external layer of transparency and the Medical Devices Framework is relevant to the insider and internal layers. We investigate the said frameworks to inform AI developers on what is already expected from them with regards to transparency. We also discover the gaps in the existing legislative frameworks concerning AI's transparency in healthcare and suggest the solutions to fill them in.

Critical review of self‐diagnosis of mental health conditions using artificial intelligence

“…The proprietary nature of AI algorithms and the commercialization of user data pose challenges related to transparency, data privacy, and security. Users may harbour scepticism about the fairness, accountability, and potential misuse of their health information, questioning the ethical underpinnings of AI and digital health technologies (Kiseleva et al., 2022; Petersson et al., 2022).…”

“…Navigating these commercial determinants requires a delicate balance between innovation and safeguarding user interests. Establishing robust regulatory frameworks becomes imperative to ensure the ethical deployment of these technologies, providing guidelines for testing, validation, and ongoing monitoring (Kiseleva et al., 2022). Transparency in algorithmic processes and proactive measures to address data privacy concerns contribute to building user trust (Kaplan, 2020).…”

Badania kliniczne wyrobów medycznych wykorzystujących inteligentne algorytmy – wstęp do dyskusji

“…szeroko znanym systemem DaVinci, który za pomocą robotycznych ramion umożliwia wykonywanie zabiegów o niespotykanej dotąd precyzji 14 . Rehabilitacja chorych po przebytych udarach mózgu, operacjach neurochirurgicznych i ortopedycznych może być wspomagana i skuteczniejsza dzięki użyciu nowoczesnych urządzeń do rehabilitacji, często działających w połączeniu z aplikacją mobilną 15 .…”

Community perspectives on AI/ML and health equity: AIM-AHEAD nationwide stakeholder listening sessions

“…Furthermore, there needs to be an increased emphasis on using ethical frameworks for AI/ ML to promote health equity and justice throughout the AI lifecycle [19,20]. Training is important to increase the general public's understanding of how these models work and demystify the "black box" nature of algorithms [21,22] Particularly in healthcare, it is imperative that AI/ML is transparent. Transparency should be achieved through sets of measures applied to algorithm development, practice, and outcome predictions [21].…”

“…Training is important to increase the general public's understanding of how these models work and demystify the "black box" nature of algorithms [21,22] Particularly in healthcare, it is imperative that AI/ML is transparent. Transparency should be achieved through sets of measures applied to algorithm development, practice, and outcome predictions [21]. Stakeholders across disciplines must hold each other accountable in balancing how public and private sectors communicate about AI, document processes, manage and govern data, and develop shared meaning around algorithmic decision making.…”