A Prospective Observational Study to Investigate Performance of a Chest X-ray Artificial Intelligence Diagnostic Support Tool Across 12 U.S. Hospitals

Abstract

Importance: An artificial intelligence (AI)-based model to predict COVID-19 likelihood from chest x-ray (CXR) findings can serve as an important adjunct to accelerate immediate clinical decision making and improve clinical decision making. Despite significant efforts, many limitations and biases exist in previously developed AI diagnostic models for COVID-19. Utilizing a large set of local and international CXR images, we developed an AI model with high performance on temporal and external validation. Objective: Investigate real-time performance of an AI-enabled COVID-19 diagnostic support system across a 12-hospital system. Design: Prospective observational study. Setting: Labeled frontal CXR images (samples of COVID-19 and non-COVID-19) from the M Health Fairview (Minnesota, USA), Valencian Region Medical ImageBank (Spain), MIMIC-CXR, Open-I 2013 Chest X-ray Collection, GitHub COVID-19 Image Data Collection (International), Indiana University (Indiana, USA), and Emory University (Georgia, USA) Participants: Internal (training, temporal, and real-time validation): 51,592 CXRs; Public: 27,424 CXRs; External (Indiana University): 10,002 CXRs; External (Emory University): 2002 CXRs Main Outcome and Measure: Model performance assessed via receiver operating characteristic (ROC), Precision-Recall curves, and F1 score. Results: Patients that were COVID-19 positive had significantly higher COVID-19 Diagnostic Scores (median .1 [IQR: 0.0-0.8] vs median 0.0 [IQR: 0.0-0.1], p < 0.001) than patients that were COVID-19 negative. Pre-implementation the AI-model performed well on temporal validation (AUROC 0.8) and external validation (AUROC 0.76 at Indiana U, AUROC 0.72 at Emory U). The model was noted to have unrealistic performance (AUROC > 0.95) using publicly available databases. Real-time model performance was unchanged over 19 weeks of implementation (AUROC 0.70). On subgroup analysis, the model had improved discrimination for patients with severe as compared to mild or moderate disease, p < 0.001. Model performance was highest in Asians and lowest in whites and similar between males and females. Conclusions and Relevance: AI-based diagnostic tools may serve as an adjunct, but not replacement, for clinical decision support of COVID-19 diagnosis, which largely hinges on exposure history, signs, and symptoms. While AI-based tools have not yet reached full diagnostic potential in COVID-19, they may still offer valuable information to clinicians taken into consideration along with clinical signs and symptoms.

Barriers to and Facilitators for Acceptance of Comprehensive Clinical Decision Support System–Driven Care Maps for Patients With Thoracic Trauma: Interview Study Among Health Care Providers and Nurses

“…AI diagnostic models using biomedical imaging are increasingly being investigated to improve diagnostic accuracy and minimize the workload of radiologists. They are used to facilitate imaging diagnosis for simple tasks and have been successfully used for several disease processes, including COVID-19 [ 50 ], acute respiratory distress syndrome [ 51 ], and pneumothorax [ 52 ] detection. We envision using a similar model for rib fracture detection.…”

Effective prognostic and clinical risk stratification in COVID‐19 using multimodality biomarkers

“…Further, the percentage opacification on CXR, as assessed visually, also performed well for predicting the risk for ITU admission and mortality [286]. Recently, the use of artificial intelligence (AI) to provide automated CXR severity scoring has received interest [294][295][296]. Some studies suggest that AI performs similarly to [294], or even better than [295], human scoring for predicting clinical outcomes [294,295].…”

“…Some studies suggest that AI performs similarly to [294], or even better than [295], human scoring for predicting clinical outcomes [294,295]. AI-based tools may become a useful future adjunct for the assessment of COVID-19 patients [296], and rigorous prospective validation is required to translate it into the forefront of clinical practice.…”

Rethinking Transfer Learning for Medical Image Classification

“…Our COVID-19 dataset comprises 3,998 COVID-positive and 41,515 COVIDnegative frontal chest x-rays collected from M Health Fairview, Minnesota, USA; details can be found in [24]. We focus on prospective evaluation: training and validation are performed on x-rays dated before June 30th 2020, and test is performed on x-rays dated after.…”