Faithful AI in Medicine: A Systematic Review with Large Language Models and Beyond

Abstract

Artificial intelligence (AI) holds great promise in healthcare and medicine on being able to help with many aspects, from biological scientific discovery, to clinical patient care, to public health policy making. However, the potential risk of AI methods for generating factually incorrect or unfaithful information is a big concern, which could result in serious consequences. This review aims to provide a comprehensive overview of the faithfulness problem in existing research on AI in healthcare and medicine, including analysis of the cause of unfaithful results, evaluation metrics, and mitigation methods. We will systematically review the recent progress in optimizing the factuality in various generative medical AI methods, including knowledge grounded large language models, text-to-text generation tasks such as medical text summarization, medical text simplification, multimodality-to-text generation tasks such as radiology report generation, and automatic medical fact-checking. The challenges and limitations of ensuring the faithfulness of AI-generated information in these applications, as well as forthcoming opportunities will be discussed. We expect this review to help researchers and practitioners understand the faithfulness problem in AI-generated information in healthcare and medicine, as well as the recent progress and challenges on related research. Our review can also serve as a guide for researchers and practitioners who are interested in applying AI in medicine and healthcare.

The Emerging Role of ISO 42001 Certification in Fostering the Deployment of Responsible Generative AI Healthcare Solutions

“…Most initial publications on the evaluation of these GenAI solutions center on evaluating unregulated summarization and translation tools 51,64 . Some other studies have focused on evaluating experimental conversational GenAI solutions, which aim to address the accuracy and contextual relevance of the information provided to patients and providers 52,65 .…”

“…The EU AI Act has extraterritorial reach and imposes numerous obligations on providers, deployers, importers, and distributors of such high-risk healthcare AI systems 29,35,[51][52][53] . The EU AI Act gradually becomes effective, with the obligations specific to high-risk AI systems going into effect 12 months after the EU AI Act enters into force, and violation of such obligations can involve penalties of up to 3% of global annual turnover (or 15 million euros, whichever is larger).…”

The Era of Artificial Intelligence Deception: Unraveling the Complexities of False Realities and Emerging Threats of Misinformation

“…Through concerted efforts to enhance the robustness of AI systems and promote transparency, the AI community can work toward mitigating the risks associated with AI hallucinations and safeguarding the integrity of AI applications. Collective insights from researchers such as [3][4][5][6][7][8][9][10][11] underscore the multifaceted nature of this issue and the diverse strategies required to address it. The potential for AI to inadvertently transcribe unsafe content on platforms such as YouTube is a stark reminder of the challenges posed by AI hallucinations.…”

Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysis

“…Several parallel research lines aim to enhance radiology report summarization with a different methodological focus. First, several studies optimize factual consistency through reinforcement learning (Zhang et al, 2020b;Delbrouck et al, 2022) or post-hoc reranking (Xie et al, 2023). Second, Karn et al ( 2022) devise an extract-then-abstract pipeline with multi-agent reinforcement learning.…”