Performance targets and the Brier score

Roulston, Mark S.

doi:10.1002/met.21

Cited by 33 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, the model with the best average prediction performance on five test datasets was considered the optimal model. We compared the developed models using five different performance measures: Brier score [44], AUROC [45], sensitivity, specificity and calibration plot. Brier score measures the overall prediction performance, which denotes the accuracy of a prediction, of a model.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

Kong

Lin

2020

BMC Med Inform Decis Mak

108

View full text Add to dashboard Cite

Background Early and accurate identification of sepsis patients with high risk of in-hospital death can help physicians in intensive care units (ICUs) make optimal clinical decisions. This study aimed to develop machine learning-based tools to predict the risk of hospital death of patients with sepsis in ICUs. Methods The source database used for model development and validation is the medical information mart for intensive care (MIMIC) III. We identified adult sepsis patients using the new sepsis definition Sepsis-3. A total of 86 predictor variables consisting of demographics, laboratory tests and comorbidities were used. We employed the least absolute shrinkage and selection operator (LASSO), random forest (RF), gradient boosting machine (GBM) and the traditional logistic regression (LR) method to develop prediction models. In addition, the prediction performance of the four developed models was evaluated and compared with that of an existent scoring tool – simplified acute physiology score (SAPS) II – using five different performance measures: the area under the receiver operating characteristic curve (AUROC), Brier score, sensitivity, specificity and calibration plot. Results The records of 16,688 sepsis patients in MIMIC III were used for model training and test. Amongst them, 2949 (17.7%) patients had in-hospital death. The average AUROCs of the LASSO, RF, GBM, LR and SAPS II models were 0.829, 0.829, 0.845, 0.833 and 0.77, respectively. The Brier scores of the LASSO, RF, GBM, LR and SAPS II models were 0.108, 0.109, 0.104, 0.107 and 0.146, respectively. The calibration plots showed that the GBM, LASSO and LR models had good calibration; the RF model underestimated high-risk patients; and SAPS II had the poorest calibration. Conclusion The machine learning-based models developed in this study had good prediction performance. Amongst them, the GBM model showed the best performance in predicting the risk of in-hospital death. It has the potential to assist physicians in the ICU to perform appropriate clinical interventions for critically ill sepsis patients and thus may help improve the prognoses of sepsis patients in the ICU.

show abstract

Section: Discussionmentioning

confidence: 99%

“…We compared the developed models using five different performance measures: Brier score [ 44 ], AUROC [ 45 ], sensitivity, specificity and calibration plot. Brier score measures the overall prediction performance, which denotes the accuracy of a prediction, of a model.…”

Section: Methodsmentioning

confidence: 99%

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

Kong

Lin

2020

BMC Med Inform Decis Mak

108

View full text Add to dashboard Cite

show abstract

“…E max refers to the maximum calibration error, and the optimal E max is close to 0. The Brier score measures the accuracy of a probability assessment (the average squared deviation between predicted probabilities for a set of events and their outcomes); the optimal Brier score is close to 0. Non‐parametric statistics were used.…”

Section: Methodsmentioning

confidence: 99%

The Peptic Ulcer Perforation (PULP) score: a predictor of mortality following peptic ulcer perforation. A cohort study

Møller

Engebjerg

Adamsen

et al. 2011

Acta Anaesthesiol Scand

110

129

View full text Add to dashboard Cite

show abstract

“…if a forecaster receives a bonus if his/her Brier score is less than a threshold, or if the forecaster with the smallest Brier score receives a bonus) then the score is rendered improper (Roulston, 2007). Consider the case of the forecaster who is offered a bonus if his/her weather forecasts over a 10-day period achieve a Brier score of less than 0.1.…”

Section: Proprietymentioning

confidence: 99%

Understanding forecast verification statistics

Mason

2008

Meteorological Applications

View full text Add to dashboard Cite

ABSTRACT:Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus on p-values and confidence intervals. P -values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out-of-sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date.

show abstract

Performance targets and the Brier score

Cited by 33 publications

References 16 publications

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

The Peptic Ulcer Perforation (PULP) score: a predictor of mortality following peptic ulcer perforation. A cohort study

Understanding forecast verification statistics

Contact Info

Product

Resources

About