Incorporating algorithmic uncertainty into a clinical machine deep learning algorithm for urgent head CTs

Yoon, Byung Chul; Pomerantz, Stuart R.; Mercaldo, Nathaniel D.; Goyal, Swati; L'Italien, Eric M; Lev, Michael H.; Buch, Karen; Buchbinder, Bradley R.; Chen, John W.; Conklin, John; Gupta, Rajiv; Hunter, George J.; Kamalian, Shahmir; Kelly, Hillary R.; Rapalino, Otto; Rincon, Sandra; Romero, Javier; He, Julian; Schaefer, Pamela W.; Do, Synho; González, R Gilberto

doi:10.1371/journal.pone.0281900

Cited by 4 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is achieved using recent representative imaging data from the local site. Specifically, we split patients into a low-risk group with a high negative predictive value, a high-risk group with a high positive predictive value, and an uncertain group, building on a previous approach developed at the Massachusetts General Hospital wherein neuroradiologists decide these groupings manually [3, 18]. In our new approach, these groupings are decided by a black-box machine learning algorithm, using imaging as input, in a way that endows the groupings with statistical guarantees.…”

Section: Introductionmentioning

confidence: 99%

“…We call this pipeline conformal triage , as the triage is determined by an AI algorithm wrapped by a conformal prediction-type procedure [17, 1, 2]. Compared to existing techniques [7, 3, 18], conformal triage is post hoc (it does not require any retraining/fine-tuning), and does not make any assumptions about the form of the model or data distribution. Thus, we can provide formal guarantees of high predictive value even if there has been drift from the original training conditions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Conformal Triage for Medical Imaging AI Deployment

Angelopoulos,

Pomerantz,

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Background: The deployment of black-box AI models in medical imaging presents significant challenges, especially in maintaining reliability across different clinical settings. These challenges are compounded by distribution shifts that can lead to failures in reproducing the accuracy attained during the AI model's original validations. Method: We introduce the conformal triage algorithm, designed to categorize patients into low-risk, high-risk, and uncertain groups within a clinical deployment setting. This method leverages a combination of a black-box AI model and conformal prediction techniques to offer statistical guarantees of predictive power for each group. The high-risk group is guaranteed to have a high positive predictive value, while the low-risk group is assured a high negative predictive value. Prediction sets are never constructed; instead, conformal techniques directly assure high accuracy in both groups, even in clinical environments different from those in which the AI model was originally trained, thereby ameliorating the challenges posed by distribution shifts. Importantly, a representative data set of exams from the testing environment is required to ensure statistical validity. Results: The algorithm was tested using a head CT model previously developed by Do and colleagues [9] and a data set from Massachusetts General Hospital. The results demonstrate that the conformal triage algorithm provides reliable predictive value guarantees to a clinically significant extent, reducing the number of false positives from 233 (45%) to 8 (5%) while only abstaining from prediction on 14% of data points, even in a setting different from the training environment of the original AI model. Conclusions: The conformal triage algorithm offers a promising solution to the challenge of deploying black-box AI models in medical imaging across varying clinical settings. By providing statistical guarantees of predictive value for categorized patient groups, this approach significantly enhances the reliability and utility of AI in optimizing medical imaging workflows, particularly in neuroradiology.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Conformal Triage for Medical Imaging AI Deployment

Angelopoulos,

Pomerantz,

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This technology can help extract advanced features and patterns from complex neural imaging data 10 . Furthermore, owing to its remarkable multidimensional analytical capabilities, machine learning can be used for classification at the individual level in the field of medical imaging 11 , 12 . However, neuroimaging-based techniques are rarely used to detect and predict FoG in PD.…”

Section: Introductionmentioning

confidence: 99%

Cortical thickness and white matter microstructure predict freezing of gait development in Parkinson’s disease

Lin,

Zou,

et al. 2024

npj Parkinsons Dis.

View full text Add to dashboard Cite

The clinical applications of the association of cortical thickness and white matter fiber with freezing of gait (FoG) are limited in patients with Parkinson’s disease (PD). In this retrospective study, using white matter fiber from diffusion-weighted imaging and cortical thickness from structural-weighted imaging of magnetic resonance imaging, we investigated whether a machine learning-based model can help assess the risk of FoG at the individual level in patients with PD. Data from the Parkinson’s Disease Progression Marker Initiative database were used as the discovery cohort, whereas those from the Fujian Medical University Union Hospital Parkinson’s Disease database were used as the external validation cohort. Clinical variables, white matter fiber, and cortical thickness were selected by random forest regression. The selected features were used to train the support vector machine(SVM) learning models. The median area under the receiver operating characteristic curve (AUC) was calculated. Model performance was validated using the external validation cohort. In the discovery cohort, 25 patients with PD were defined as FoG converters (15 men, mean age 62.1 years), whereas 60 were defined as FoG nonconverters (38 men, mean age 58.5 years). In the external validation cohort, 18 patients with PD were defined as FoG converters (8 men, mean age 66.9 years), whereas 37 were defined as FoG nonconverters (21 men, mean age 65.1 years). In the discovery cohort, the model trained with clinical variables, cortical thickness, and white matter fiber exhibited better performance (AUC, 0.67–0.88). More importantly, SVM-radial kernel models trained using random over-sampling examples, incorporating white matter fiber, cortical thickness, and clinical variables exhibited better performance (AUC, 0.88). This model trained using the above mentioned features was successfully validated in an external validation cohort (AUC, 0.91). Furthermore, the following minimal feature sets that were used: fractional anisotropy value and mean diffusivity value for right thalamic radiation, age at baseline, and cortical thickness for left precentral gyrus and right dorsal posterior cingulate gyrus. Therefore, machine learning-based models using white matter fiber and cortical thickness can help predict the risk of FoG conversion at the individual level in patients with PD, with improved performance when combined with clinical variables.

show abstract