Resampling and cross-validation techniques: a tool to reduce bias caused by model building?

Schumacher, Martin; Holländer, Norbert; Sauerbrei, Willi

doi:10.1002/(sici)1097-0258(19971230)16:24<2813::aid-sim701>3.0.co;2-z

Cited by 135 publications

(81 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This type of effect is expected, because stepwise selection procedures during model derivation tend to overfit the model to the derivation data, thus leading to optimistic C statistics. 27 In the validation models with 15, 8, and then 5 items, the observed decreases in the C statistics demonstrate why such models need to be validated in distinct patient populations. Another part of the difference in the predictive performance of the derivation versus validation models may have to do with differences in the patient populations.…”

Section: Discussionmentioning

confidence: 99%

Shortening the NIH Stroke Scale for Use in the Prehospital Setting

Tirschwell

Longstreth

Becker

et al. 2002

Stroke

View full text Add to dashboard Cite

Background and Purpose— Prehospital stroke scales should identify stroke patients and measure stroke severity. The goal of this study was to identify a subset of the 15 items in the National Institutes of Health Stroke Scale (NIHSS-15) that measures stroke severity and predicts outcomes. Methods— Using 2 distinct data sets from acute stroke clinical trials, we derived and validated shortened versions of the NIHSS (sNIHSS). Stepwise logistic regression and bootstrap techniques were used in selection of NIHSS-15 items. Areas under the receiver operator characteristic curve (C statistics) were used to compare predictive performance of logistic models incorporating differing versions of the NIHSS. Results— The derivation analyses suggested the 8 NIHSS-15 items that were most predictive of “good outcome” 3 months after stroke, in order of decreasing importance: right leg item, left leg, gaze, visual fields, language, level of consciousness, facial palsy, and dysarthria. The sNIHSS-8 comprises all 8 and the sNIHSS-5, the first 5. In the validation models, C statistics were NIHSS-15=0.80, sNIHSS-8=0.77, and sNIHSS-5=0.76. Statistical comparisons suggested that the NIHSS-15 had better predictive performance than the sNIHSS-8 or the sNIHSS-5; the absolute difference in C statistics was small. There was no significant difference between the sNIHSS-8 and the sNIHSS-5. Conclusions— Much of the predictive performance of the full NIHSS-15 was retained with a shortened scale, the sNIHSS-5. Shortening the NIHSS-15 will facilitate its use during prehospital evaluations. The sNIHSS severity information may be useful to triage acute stroke patients in communities and to provide a baseline stroke severity for prehospital acute stroke trials.

show abstract

Section: Discussionmentioning

confidence: 99%

Shortening the NIH Stroke Scale for Use in the Prehospital Setting

Tirschwell

Longstreth

Becker

et al. 2002

Stroke

View full text Add to dashboard Cite

show abstract

“…Bootstrapping techniques were used for internal validation of the model (26,27), and bootstrap samples were drawn 200 times with replacement. Regression models were created in each bootstrap sample and tested on the original sample to obtain stable estimates of the optimism of the model, i.e., how much the model performance was expected to decrease when applied in new datasets (28)(29)(30). All analyses were performed using STATA version 13.1 (StataCorp LP) and SAS statistical software version 9.3 (SAS Institute).…”

Section: Statistical Analysesmentioning

confidence: 99%

LLPi: Liverpool Lung Project Risk Prediction Model for Lung Cancer Incidence

Marcus

Chen

Raji

et al. 2015

Cancer Prevention Research

View full text Add to dashboard Cite

Identification of high-risk individuals will facilitate early diagnosis, reduce overall costs, and also improve the current poor survival from lung cancer. The Liverpool Lung Project prospective cohort of 8,760 participants ages 45 to 79 years, recruited between 1998 and 2008, was followed annually through the hospital episode statistics until January 31, 2013. Cox proportional hazards models were used to identify risk predictors of lung cancer incidence. C-statistic was used to assess the discriminatory accuracy of the models. Models were internally validated using the bootstrap method. During mean follow-up of 8.7 years, 237 participants developed lung cancer. Age [hazard ratio (HR), 1.04; 95% confidence interval (CI), 1.02-1.06], male gender (HR, 1.48; 95% CI, 1.10-1.98), smoking duration (HR, 1.04; 95% CI, 1.03-1.05), chronic obstructive pulmonary disease (HR, 2.43; 95% CI, 1.79-3.30), prior diagnosis of malignant tumor (HR, 2.84; 95% CI, 2.08-3.89), and early onset of family history of lung cancer (HR, 1.68; 95% CI, 1.04-2.72) were associated with the incidence of lung cancer. The LLPi risk model had a good calibration (goodness-of-fit c 2 7.58, P ¼ 0.371). The apparent C-statistic was 0.852 (95% CI, 0.831-0.873) and the optimism-corrected bootstrap resampling C-statistic was 0.849 (95% CI, 0.829-0.873). The LLPi risk model may assist in identifying individuals at high risk of developing lung cancer in population-based screening programs. Cancer Prev Res; 8(6); 570-5. Ó2015 AACR.

show abstract

“…11 Other, less common model approaches resort to complex mathematical analytics of the data. These models often utilize a broad range of methods involving machine learning and pattern recognition, among others, 12,13 and they are often, but not always, limited to classification tree, neural network, k-nearest neighbor. 13 The model is often trained on large number of individuals of the cohort and validated on a faction of the cohort data or on data from another study.…”

Section: Predictive Modelsmentioning

confidence: 99%

Toward Big Data Analytics

Cichosz

Johansen

Hejlesen

2015

J Diabetes Sci Technol

View full text Add to dashboard Cite

Diabetes is one of the top priorities in medical science and health care management, and an abundance of data and information is available on these patients. Whether data stem from statistical models or complex pattern recognition models, they may be fused into predictive models that combine patient information and prognostic outcome results. Such knowledge could be used in clinical decision support, disease surveillance, and public health management to improve patient care. Our aim was to review the literature and give an introduction to predictive models in screening for and the management of prevalent short-and long-term complications in diabetes. Predictive models have been developed for management of diabetes and its complications, and the number of publications on such models has been growing over the past decade. Often multiple logistic or a similar linear regression is used for prediction model development, possibly owing to its transparent functionality. Ultimately, for prediction models to prove useful, they must demonstrate impact, namely, their use must generate better patient outcomes. Although extensive effort has been put in to building these predictive models, there is a remarkable scarcity of impact studies.

show abstract

Resampling and cross-validation techniques: a tool to reduce bias caused by model building?

Cited by 135 publications

References 12 publications

Shortening the NIH Stroke Scale for Use in the Prehospital Setting

Shortening the NIH Stroke Scale for Use in the Prehospital Setting

LLPi: Liverpool Lung Project Risk Prediction Model for Lung Cancer Incidence

Toward Big Data Analytics

Contact Info

Product

Resources

About