The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases

Ozenne, Brice; Subtil, Fabien; Maucort-Boulch, Delphine

doi:10.1016/j.jclinepi.2015.02.010

Cited by 270 publications

(196 citation statements)

References 11 publications

Supporting

Mentioning

192

Contrasting

Unclassified

Order By: Relevance

“…AUROC is one of the most used metrics and shows sensitivity against 1−specificity. Compared with AUROC, AUPRC is suitable for verifying false‐alarm rates with varying sensitivity and shows precision (ie, 1−false‐alarm rate) against recall (ie, sensitivity) 34, 35…”

Section: Methodsmentioning

confidence: 99%

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

Kwon

Lee²,

Lee³

et al. 2018

JAHA

234

205

View full text Add to dashboard Cite

BackgroundIn‐hospital cardiac arrest is a major burden to public health, which affects patient safety. Although traditional track‐and‐trigger systems are used to predict cardiac arrest early, they have limitations, with low sensitivity and high false‐alarm rates. We propose a deep learning–based early warning system that shows higher performance than the existing track‐and‐trigger systems.Methods and ResultsThis retrospective cohort study reviewed patients who were admitted to 2 hospitals from June 2010 to July 2017. A total of 52 131 patients were included. Specifically, a recurrent neural network was trained using data from June 2010 to January 2017. The result was tested using the data from February to July 2017. The primary outcome was cardiac arrest, and the secondary outcome was death without attempted resuscitation. As comparative measures, we used the area under the receiver operating characteristic curve (AUROC), the area under the precision–recall curve (AUPRC), and the net reclassification index. Furthermore, we evaluated sensitivity while varying the number of alarms. The deep learning–based early warning system (AUROC: 0.850; AUPRC: 0.044) significantly outperformed a modified early warning score (AUROC: 0.603; AUPRC: 0.003), a random forest algorithm (AUROC: 0.780; AUPRC: 0.014), and logistic regression (AUROC: 0.613; AUPRC: 0.007). Furthermore, the deep learning–based early warning system reduced the number of alarms by 82.2%, 13.5%, and 42.1% compared with the modified early warning system, random forest, and logistic regression, respectively, at the same sensitivity.ConclusionsAn algorithm based on deep learning had high sensitivity and a low false‐alarm rate for detection of patients with cardiac arrest in the multicenter study.

show abstract

Section: Methodsmentioning

confidence: 99%

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

Kwon

Lee²,

Lee³

et al. 2018

JAHA

234

205

View full text Add to dashboard Cite

show abstract

“…AUCs are bounded by 0 and 1, where in general an AUC close to 1 indicates a nearly perfect prediction, an AUC of 0.5 indicates that the prediction is no better than chance, and an AUC of less than 0.5 indicates worse prediction than by chance. For situations in which the prevalence of an event is to be accounted for, the metrics of positive predicted value (PPV), negative predicted value (NPV), see e.g., Kuhn & Johnson 16 , and related measures (e.g., Ozenne et al 27 ) are available beyond sensitivity, specificity, and the ROC.…”

Section: Methodsmentioning

confidence: 99%

The importance of prediction model validation and assessment in obesity and nutrition research

Ivanescu

George

et al. 2015

Int J Obes

View full text Add to dashboard Cite

Deriving statistical models to predict one variable from one or more other variables, or predictive modeling, is an important activity in obesity and nutrition research. To determine the quality of the model, it is necessary to quantify and report the predictive validity of the derived models. Conducting validation of the predictive measures provides essential information to the research community about the model. Unfortunately, many articles fail to account for the nearly inevitable reduction in predictive ability that occurs when a model derived on one dataset is applied to a new dataset. Under some circumstances, the predictive validity can be reduced to nearly zero. In this overview, we explain why reductions in predictive validity occur, define the metrics commonly used to estimate the predictive validity of a model (e.g., R2, mean squared error, sensitivity, specificity, receiver operating characteristic, concordance index), and describe methods to estimate the predictive validity (e.g., cross-validation, bootstrap, adjusted and shrunken R2). We emphasize that methods for estimating the expected reduction in predictive ability of a model in new samples are available and this expected reduction should always be reported when new predictive models are introduced.

show abstract

“…19,20 We performed subgroup analyses of CHD (ICD-10 codes: I20-I25) and HF (ICD-10 codes: I50 and I255) with hospital B patients. The AUROC is one of the most commonly used metrics for prediction model and shows sensitivity against 1-specificity.…”

Section: Me Thodsmentioning

confidence: 99%

“…tion data. This study validated that DL showed the best performance in other situations through external validation and subgroup analysis.With imbalanced data, in which the number of negatives outweighs the number of positives, the AUROC has a limitation for evaluating the performance because the false positivity rate (number of false positives/total number of real negatives) does not decrease dramatically when the total number of negatives is large 19. Wolpert explains the "no free lunch theorem"; if optimized in one situation, a model cannot produce good results in other situations 22.…”

mentioning

confidence: 99%

Deep learning for predicting in‐hospital mortality among heart disease patients based on echocardiography

et al. 2018

View full text Add to dashboard Cite

Background: Heart disease (HD) is the leading cause of global death; there are several mortality prediction models of HD for identifying critically-ill patients and for guiding decision making. The existing models, however, cannot be used during initial treatment or screening. This study aimed to derive and validate an echocardiographybased mortality prediction model for HD using deep learning (DL). Methods:In this multicenter retrospective cohort study, the subjects were admitted adult (age ≥ 18 years) HD patients who underwent echocardiography. The outcome was in-hospital mortality. We extracted predictor variables from echocardiography reports using text mining. We developed deep learning-based prediction model using derivation data of a hospital A. And we conducted external validation using echocardiography report of hospital B. We conducted subgroup analysis of coronary heart disease (CHD) and heart failure (HF) patients of hospital B and compared DL with the currently used predictive models (eg, Global Registry of Acute Coronary Events (GRACE) score, Thrombolysis in Myocardial Infarction score (TIMI), Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC) score, and Get With The Guidelines-Heart Failure (GWTG-HF) score). Results: The study subjects comprised 25 776 patients with 1026 mortalities. The areas under the receiver operating characteristic curve (AUROC) of the DL model were 0.912, 0.898, 0.958, and 0.913 for internal validation, external validation, CHD, and HF, respectively, and these results significantly outperformed other comparison models.Conclusions: This echocardiography-based deep learning model predicted in-hospital mortality among HD patients more accurately than existing prediction models and other machine learning models.

show abstract

The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases

Cited by 270 publications

References 11 publications

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

The importance of prediction model validation and assessment in obesity and nutrition research

Deep learning for predicting in‐hospital mortality among heart disease patients based on echocardiography

Contact Info

Product

Resources

About