Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Abstract

he prospect of improved clinical outcomes and more efficient health systems has fueled a rapid rise in the development and evaluation of AI systems over the last decade. Because most AI systems within healthcare are complex interventions designed as clinical decision support systems, rather than autonomous agents, the interactions among the AI systems, their users and the implementation environments are defining components of the AI interventions' overall potential effectiveness. Therefore, bringing AI systems from mathematical performance to clinical utility needs an adapted, stepwise implementation and evaluation pathway, addressing the complexity of this collaboration between two independent forms of intelligence, beyond measures of effectiveness alone 1 . Despite indications that some AI-based algorithms now match the accuracy of human experts within preclinical in silico studies 2 , there

A systematic review of radiomics in pancreatitis: applying the evidence level rating tool for promoting clinical transferability

“…During this stage, the researchers, authors, reviewers, and editors of radiomics studies play an important role to improve the methodological and reporting quality, and make sure only studies with adequate innovation are being published. Next, at the stage of safety and utility, the small-scale early live clinical evaluations are used to inform regulatory decisions and are part of the clinical evidence generation process [ 65 ]. With improvements of study quality, the radiomics research community could for the first time provide more robust scientific evidence for the translation of radiomics.…”

“…Indeed, the IBSI guideline used in our review is an achievement gained by an independent international collaboration which works towards standardization of the radiomics methodology and reporting [ 11 ]. There are many other guidelines developed or under development by the radiomics and artificial intelligence community with the purpose to improve study quality, including Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis based on Artificial Intelligence (TRIPOD-AI) [ 63 ], Prediction model Risk Of Bias ASsessment Tool based on Artificial Intelligence (PROBAST-AI) [ 63 ], Quality Assessment of Diagnostic Accuracy Studies centered on Artificial Intelligence (QUADAS-AI) [ 64 ], Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI) [ 65 ], Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence (SPRIIT-AI) [ 66 ], Consolidated Standards of Reporting Trials–Artificial Intelligence (CONSORT-AI) [ 67 ], Standards for Reporting of Diagnostic Accuracy Study centered on Artificial Intelligence (STARD-AI) [ 68 ], Checklist for Artificial Intelligence in Medical Imaging (CLAIM) [ 69 ], etc. Their project teams and steering committees usually consisted of a broad range of experts to provide balanced and diverse views involving various stakeholder groups.…”

Bias in Machine Learning Models Can Be Significantly Mitigated by Careful Training: Evidence from Neuroimaging Studies

“…As machine learning models are being applied to diverse problems in the clinical sciences, there is an increasing amount of discussion on bias in particular, and ethical issues in general 21,23,[33][34][35] . This literature is playing a crucial role in shaping policies for deployment of automated diagnostic models.…”

Relationship Between Prediction Accuracy and Feature Importance Reliability: an Empirical and Theoretical Study

“…This ambition requires users to have trust in the predictive models, which often rests on a given models' interpretability (Bussone et al 2015, Price 2018, Anderson and Anderson 2019, Diprose et al 2020, Hedderich and Eickhoff 2020. Indeed, the recently enacted European Union Global Data Protection Regulation (GDPR) states that patients have a right to "meaningful information about the logic involved" when automated decision-making systems are used (Vasey et al 2022a(Vasey et al , 2022b. Furthermore, in many studies, the derived predictive models are often interpreted to gain insights into the predictive principles and inter-individual differences that underpin observed brain-behavior relationships (Finn et al 2015, Greene et al 2018).…”