The Deception of Certainty: how Non-Interpretable Machine Learning Outcomes Challenge the Epistemic Authority of Physicians. A deliberative-relational Approach

Abstract

Developments in Machine Learning (ML) have attracted attention in a wide range of healthcare fields to improve medical practice and the benefit of patients. Particularly, this should be achieved by providing more or less automated decision recommendations to the treating physician. However, some hopes placed in ML for healthcare seem to be disappointed, at least in part, by a lack of transparency or traceability. Skepticism exists primarily in the fact that the physician, as the person responsible for diagnosis, therapy, and care, has no or insufficient insight into how such recommendations are reached. The following paper aims to make understandable the specificity of the deliberative model of a physician-patient relationship that has been achieved over decades. By outlining the (social-)epistemic and inherently normative relationship between physicians and patients, I want to show how this relationship might be altered by non-traceable ML recommendations. With respect to some healthcare decisions, such changes in deliberative practice may create normatively far-reaching challenges. Therefore, in the future, a differentiation of decision-making situations in healthcare with respect to the necessary depth of insight into the process of outcome generation seems essential.

Physician’s autonomy in the face of AI support: walking the ethical tightrope

“…In the face of AI support, this is changing in that the form of data evaluation is taking place in a new way, one that is difficult for humans to comprehend, and it is moving alongside the methods already used to achieve action-guiding knowledge (e.g., guidelines from medical professional societies, medical-theoretical expertise). Particularly in the case of divergent or disagreeing action-guiding knowledge, decision-making situations can arise that are difficult to resolve from a human point of view without providing underlying reasoning ( 14 , 30 , 31 ), one of the central empirically identified barriers to the use of AI support [cf. ( 8 , 9 , 32 )].…”

“…This necessary information varies in different fields of application, for example due to the consequences of a clinical decision on the patient’s life [cf. for this elsewhere ( 14 )]. In such a way, a recommendation can be made for the individual patient that best corresponds to his or her well-being and avoids the inherent shortcomings of AI support tools.…”

“…For example, if the relevant medical evidence-based guideline recommends a certain treatment as a priority that does not coincide with the recommendation of the AI supports, the physician may find herself or himself in a position where it is difficult to resolve the disagreement underlying these recommendations. This challenge is mainly discussed in the literature under the term “peer disagreement” ( 14 , 30 , 31 ). If the physician does not find himself or herself capable of resolving such a disagreement, he or she may see only the option of disclosing the divergent recommendations to the patient so that he or she can select his or her preferred form of knowledge reasoning.…”

“…Preliminary empirical results of the past years, especially from qualitative interview studies with doctors, underline the relevance of the topic of physician’s autonomy and related concepts like responsibility, control, and decision-making authority ( 6–13 ). According to the arguments put forward by interviewees ( 7 , 8 , 10 ) as well as within theory-building ( 2 , 14 ), there are potential risks to the physician’s autonomy such as “de-skilling,” i.e., the gradual loss through non-use or reduced relevance of the use of certain skills, or the inability to provide adequate information about risks and possible errors, for example due to biases. The interdisciplinary discussion of recent years has led to the widely shared consensus that AI should only be used, at least for health care of individuals, in such a way that it supports human decision-making, but in no way replaces it ( 15 ).…”

Faktoren für die Implementierung von KI-basierten Entscheidungsunterstützungssystemen zur Antibiotikavorhersage im Krankenhaus – eine qualitative Analyse aus der Perspektive von ärztlichem Personal

“…Im Gegensatz zu herkömmlichen Expertensystemen, bei denen Algorithmen die ihnen zugeführten Daten regelbasiert, d. h. anhand von deterministischen Wenn-Dann-Regeln auf die immer gleiche und damit komplexe, aber vorhersehbare Art und Weise, berechnen, können nicht-regelbasierte Algorithmen sich allein auf Basis der ihnen zugeführten Daten „eigenständig“ (weiter-)entwickeln 10 . Potenziell problematisch an solchen Anwendungen ist, dass ihre Entscheidungsstruktur nicht oder nur teilweise und unter großem Aufwand von Menschen verstanden und bewertet werden kann 11 12 .…”