Missing data was handled inconsistently in UK prediction models: a review of method used

Tsvetanova, Antonia; Sperrin, Matthew; Peek, Niels; Buchan, Iain; Hyland, Stephanie L.; Martin, Glen P.

doi:10.1016/j.jclinepi.2021.09.008

Cited by 30 publications

(14 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We performed imputation to account for missing data with regard to age, sex, telephone triage level, and doctor’s assessment using the k -nearest neighbours algorithm [ 16 ]. The following covariates were used for the imputation: age, sex, chief complaint, comorbidities and telephone triage level.…”

Section: Methodsmentioning

confidence: 99%

Machine learning models predicting undertriage in telephone triage

et al. 2022

View full text Add to dashboard Cite

Background Undertriaged patients have worse outcomes than appropriately triaged patients. Machine learning provides better triage prediction than conventional triage in emergency departments, but no machine learning-based undertriage prediction models have yet been developed for prehospital telephone triage. We developed and validated machine learning models for telephone triage. Materials and methods We conducted a retrospective cohort study with the largest after-hour house-call (AHHC) service dataset in Japan. Participants were ≥16 years and used the AHHC service between 1 November 2018 and 31 January 2021. We developed five prediction models based on support vector machine (SVM), lasso regression (LR), random forest (RF), gradient-boosted decision tree (XGB), and deep neural network (DNN). The primary outcome was undertriage, and predictors were telephone triage level and routinely available telephone-based data, including age, sex, 80 chief complaint categories and 10 comorbidities. We measured the area under the receiver operating characteristic curve (AUROC) for all the models. Results We identified 15,442 eligible patients (age: 38.4 ± 16.6, male: 57.2%), including 298 (1.9%; age: 58.2 ± 23.9, male: 55.0%) undertriaged patients. RF and XGB outperformed the other models, with the AUROC values (95% confidence interval; 95% CI) of the SVM, LR, RF, XGB and DNN for undertriage being 0.62 (0.55–0.69), 0.79 (0.74–0.83), 0.81 (0.76–0.86), 0.80 (0.75–0.84) and 0.77 (0.73–0.82), respectively. Conclusions We found that RF and XGB outperformed other models. Our findings suggest that machine learning models can facilitate the early detection of undertriage and early intervention to yield substantially improved patient outcomes. KEY MESSAGES Undertriaged patients experience worse outcomes than appropriately triaged patients; thus, we developed machine learning models for predicting undertriage in the prehospital setting. In addition, we identified the predictors of risk factors associated with undertriage. Random forest and gradient-boosted decision tree models demonstrated better prediction performance, and the models identified the risk factors associated with undertriage. Machine learning models aid in the early detection of undertriage, leading to significantly improved patient outcomes and identifying undertriage-associated risk factors, including chief complaint categories, could help prioritize conventional telephone triage protocol revision.

show abstract

Section: Methodsmentioning

confidence: 99%

Machine learning models predicting undertriage in telephone triage

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Ideally, model validation should follow the same steps as model deployment in order to properly quantify how the model will perform in practice. 3 However, since validation is usually completed for a large cohort of individuals at once (as opposed to a single individual), it is likely that missing data imputation would take place as a completely separate exercise, with the imputation model depending solely on the validation data.…”

Section: Missing Data Handling Strategiesmentioning

confidence: 99%

“…2 A common challenge in the development, validation and deployment of CPMs is the handling of missing data on predictors and outcome data. The most commonly used methods to handle missing data in CPM development and validation are complete case analysis or multiple imputation (MI) approaches, 1,3 the latter of which is often heralded as the gold standard in handling missing data. Much of the past research around this topic has focused on the performance of different imputation methods to recover unbiased parameter estimates, for example, in causal inference or hypothesis testing.…”

Section: Introductionmentioning

confidence: 99%

“…Ideally, all predictors considered for inclusion in a CPM should be either readily available, or easily measured, at the point of prediction. There exist, however, notable examples that allow missingness at the point of prediction 3 ; the QRisk3 10 and QKidney 11 algorithms are examples of such models that allow users to make a prediction in the absence of clinical predictors (such as cholesterol) that may not be available, or easily measured, at the time of prediction.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study

Sisk

Sperrin

Peek

et al. 2023

Stat Methods Med Res

Self Cite

View full text Add to dashboard Cite

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods : We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results : With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

show abstract

“…Above methods, however, have following problems. Firstly, deleting method is not applicable to the dataset who doesn't contain su cent sample size after deleting 13 . Secondly, imputing xed values will bring potential bias or unrealistic results on high-dimensional dataset 14 .…”

Section: Introductionmentioning

confidence: 99%

A novel method for handling missing data in health care real-world study: Optimal Intact Subset Method

Chang

Tong

et al. 2022

Preprint

View full text Add to dashboard Cite

Handling missing data is indispensable in health-care real-world data processing. Imputing method may introduce error and multicollinearity. Therefore, we explored (Optimal Intact Subset Method, OIS.Method) to avoid the issues. By exploring an optimal deleting way of columns and rows with missing data, a subset retaining most information of original datasets was determined. Traditionally, we can traverse all deleting ways. But the computational cost is too high to use in large datasets. OIS.Method used an indicator to determine the optimal deleting order which can ascertain the optimal deleting way and simplify computing. In order to validate the effectiveness of OIS.Method, we compared OIS.Method with five other missing data handling methods in simulated real-world classification datasets. Additionally, we validated OIS.Method in two real-world classification tasks. In simulated datasets, the performance of OIS.Method was best(highest AUC was 1). In real-world datasets, OIS.Method could acquire better classification performance. Take AUC for an example: OIS.Method VS Simple Impute VS Random Forest VS Modified Random Forest, 0.8179±0.0005 VS 0.8116±0.0002 VS 0.8087±0.0009 VS 0.8093±0.0014 in task1, and 0.7028±0.0126 VS 0.6963±0.0231 VS 0.6957±0.0247 VS 0.6699±0.0249 in task2. The calculation of OIS.Method is smaller, and it is well-suited for large real-world datasets.

show abstract

Missing data was handled inconsistently in UK prediction models: a review of method used

Cited by 30 publications

References 42 publications

Machine learning models predicting undertriage in telephone triage

Machine learning models predicting undertriage in telephone triage

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study

A novel method for handling missing data in health care real-world study: Optimal Intact Subset Method

Contact Info

Product

Resources

About