APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

Kwong, Jethro C. C.; Khondker, Adree; Lajkosz, Katherine; McDermott, Matthew B. A.; Frigola, Xavier Borrat; McCradden, Melissa D.; Mamdani, Muhammad; Kulkarni, Girish S.; Johnson, Alistair E. W.

doi:10.1001/jamanetworkopen.2023.35377

Cited by 23 publications

(3 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide the modified PROBAST in the Supplementary Materials ; this may be useful to other researchers assessing QA systems. Other AI-focused tools (eg, APPRAISE-AI 23 ) are rapidly becoming available; they cover similar aspects of bias to PROBAST.…”

Section: Methodsmentioning

confidence: 99%

Question answering systems for health professionals at the point of care—a systematic review

Kell,

Roberts,

Umansky

et al. 2024

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objectives Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement. Materials and methods We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology, and forward and backward citations on February 7, 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems. Results We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians’ questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians. Discussion While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.

show abstract

Section: Methodsmentioning

confidence: 99%

Question answering systems for health professionals at the point of care—a systematic review

Kell,

Roberts,

Umansky

et al. 2024

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

show abstract

“…To understand the quality of the current relevant research, we assessed the quality of the included studies using the APPRAISE-AI Tool, which was released on JAMA Network Open in 2023, and the Prediction model Risk Of Bias Assessment Tool (PROBAST) [ 30 , 31 ] ( Supplemental Data Tables 2 and 3 ).…”

Section: Methodsmentioning

confidence: 99%

Predictive models of sepsis-associated acute kidney injury based on machine learning: a scoping review

Li,

Zhu,

Yan

2024

Renal Failure

View full text Add to dashboard Cite

Background With the development of artificial intelligence, the application of machine learning to develop predictive models for sepsis-associated acute kidney injury has made potential breakthroughs in early identification, grading, diagnosis, and prognosis determination. Methods Here, we conducted a systematic search of the PubMed, Cochrane Library, Embase (Ovid), Web of Science, and Scopus databases on April 28, 2023, and screened relevant literature. Then, we comprehensively extracted relevant data related to machine learning algorithms, predictors, and predicted objectives. We subsequently performed a critical evaluation of research quality, data aggregation, and analyses. Results We screened 25 studies on predictive models for sepsis-associated acute kidney injury from a total of originally identified 2898 studies. The most commonly used machine learning algorithm is traditional logistic regression, followed by eXtreme gradient boosting. We categorized these predictive models into early identification models (60%), prognostic prediction models (32%), and subtype identification models (8%) according to their predictive purpose. The five most commonly used predictors were serum creatinine levels, lactate levels, age, blood urea nitrogen concentration, and diabetes mellitus. In addition, a single data source, insufficient assessment of clinical utility, lack of model bias assessment, and hyperparameter adjustment may be the main reasons for the low quality of the current research. Conclusions However, studies on the nondeath prognostic outcomes, the long-term clinical outcomes, and the subtype identification models are insufficient. Additionally, the poor quality of the research and the insufficient practicality of the model are problems that need to be addressed urgently.

show abstract

“…Higher scores indicate stronger methodological or reporting quality. This point system allows for a detailed and nuanced assessment of each study, focusing on different aspects critical for the validity and reliability of AI research in the medical field (10).…”

Section: Quality Assessment and Risk Of Biasmentioning

confidence: 99%

Artificial intelligence for predicting response to neoadjuvant chemotherapy for bladder cancer

Suartz,

Martinez,

Cordeiro

et al. 2024

CUAJ

View full text Add to dashboard Cite

Introduction: Neoadjuvant cisplatin-based combination chemotherapy (NAC) followed by radical cystectomy is the standard of care for cisplatin-fit patients harboring muscle-invasive bladder cancer (MIBC). Prediction of response to NAC is essential for clinical decision-making regarding alternatives in case of non-response and bladder-sparing in case of complete response. This research aimed to assess the performance of machine learning in predicting therapeutic response following NAC treatment in patients with MIBC. Methods: A systematic review adhering to the PRISMA guidelines was conducted until July 2023. The study integrated articles relating to artificial intelligence and NAC response in MIBC from various databases. The quality of articles was evaluated using the Quality Assessment Tool for Diagnostic Accuracy Studies 2 (QUADAS-2). A meta-analysis was subsequently performed on selected studies to determine the sensitivity and specificity of machine learning algorithms in predicting NAC response. Results: Of 655 articles identified, 12 studies comprising 1523 patients were included, and four studies were eligible for meta-analysis. The sensitivity and specificity of the studies were 0.62 (95% confidence interval [CI] 0.50–0.72) and 0.82 (95% CI 0.72–0.89), respectively, with a heterogeneity score (I2) of 38.5%. The machine learning algorithms used computed tomography, genetic, and anatomopathological data as input and exhibited promising potential for predicting NAC response. Conclusions: Machine-learning algorithms, especially those using computed tomography, genetic, and pathologic data, demonstrate significant potential for predicting NAC response in MIBC. Standardization of methodologic data analysis and response criteria are needed as validation studies.

show abstract

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

Cited by 23 publications

References 39 publications

Question answering systems for health professionals at the point of care—a systematic review

Question answering systems for health professionals at the point of care—a systematic review

Predictive models of sepsis-associated acute kidney injury based on machine learning: a scoping review

Artificial intelligence for predicting response to neoadjuvant chemotherapy for bladder cancer

Contact Info

Product

Resources

About