Randomized Controlled Trials Evaluating AI in Clinical Practice: A Scoping Evaluation

Abstract

BackgroundArtificial intelligence (AI) has emerged as a promising tool in healthcare, with numerous studies indicating its potential to perform as well or better than clinicians. However, a considerable portion of these AI models have only been tested retrospectively, raising concerns about their true effectiveness and potential risks in real-world clinical settings.MethodsWe conducted a systematic search for randomized controlled trials (RCTs) involving AI algorithms used in various clinical practice fields and locations, published between January 1, 2018, and August 18, 2023. Our study included 84 trials and focused specifically on evaluating intervention characteristics, study endpoints, and trial outcomes, including the potential of AI to improve care management, patient behavior and symptoms, and clinical decision-making.ResultsOur analysis revealed that 82·1% (69/84) of trials reported positive results for their primary endpoint, highlighting AI’s potential to enhance various aspects of healthcare. Trials predominantly evaluated deep learning systems for medical imaging and were conducted in single-center settings. The US and China had the most trials, with gastroenterology being the most common field of study. However, we also identified areas requiring further research, such as multi-center trials and diverse outcome measures, to better understand AI’s true impact and limitations in healthcare.ConclusionThe existing landscape of RCTs on AI in clinical practice demonstrates an expanding interest in applying AI across a range of fields and locations. While most trials report positive outcomes, more comprehensive research, including multi-center trials and diverse outcome measures, is essential to fully understand AI’s impact and limitations in healthcare.

Artificial intelligence in surgery

“…The use of AI in surgical risk prediction remains an emerging field that is lacking in randomized trials 55 and external validation, and has a high risk of bias 56 . Future work should move toward predictions of relevance to clinicians and patients 57 and prioritize compliance with the CONSORT-AI extension 58 , TRIPOD (and its upcoming AI extension) 59,60 , DECIDE-AI 61 and PRISMA AI 62 and other relevant reporting guidelines, to advance the field in a standardized, safe and efficient manner while minimizing research waste.…”

“…A sector of medical practice that once lagged behind in terms of evidence-based medicine 157 , surgery has evolved to thrive on world-class research and evidence 158 . Surgery now equals other fields, such as cardiology, in terms of the quantity of randomized trials in AI applications, only lagging behind frontrunner fields such as gastroenterology and radiology, where task-specific applications are opportune, particularly around image processing 55 . In this Review, we have highlighted many of the most pragmatic and innovative emerging use cases of AI in surgery, with a particular focus on direct feasibility and preparedness for clinical translation, but there remain numerous additional examples and untapped avenues for further pursuit.…”

“…In the sections above, we present several opportunities for such generalizable AI models unburdened by labeling needs to be implemented in surgical care as the generalist AI surgeon augmenter. These approaches are common to AI in medicine, with the majority of approaches using decision trees, neural networks and reinforcement learning 55 . Early implementations of existing LLMs for text generation, data extraction and patient care are undoubtedly underway 163 , with notable caveats such as model accuracy degradation, output overconfidence, lack of data privacy and regulatory approvals and a deficiency of prospective clinical trials yet to be overcome.…”

“…AI in surgery is a rapidly developing and promising avenue for innovation; the realization of this potential will be underpinned by increased collaboration 154 , robust randomized trial evidence 55 , the exploration of novel use cases 164 and the development of a digitally minded surgical infrastructure to enable this technological transformation. The role of AI in surgery is set to expand dramatically and, with correct oversight, its ultimate promise is to effectively improve both patient and operator outcomes, reduce patient morbidity and mortality and enhance the delivery of surgery globally.…”

From Theoretical Models to Practical Deployment: A Perspective and Case Study of Opportunities and Challenges in AI-driven Cardiac Auscultation Research for Low-Income Settings

“…However, these trials have only been conducted in a few dozen studies [27]. One study, for example, used a randomised trail to investigate performance, costs, and especially treatment time for HIV-Tuberculosis screening in Malawi [28].…”