How handling missing data may impact conclusions: A comparison of six different imputation methods for categorical questionnaire data

Abstract

Objectives:Missing data is a recurrent issue in many fields of medical research, particularly in questionnaires. The aim of this article is to describe and compare six conceptually different multiple imputation methods, alongside the commonly used complete case analysis, and to explore whether the choice of methodology for handling missing data might impact clinical conclusions drawn from a regression model when data are categorical.Methods:In addition to the commonly used complete case analysis, we tested the following six imputation methods: multiple imputation using expectation–maximization with bootstrapping, multiple imputation using multiple correspondence analysis, multiple imputation using latent class analysis, multiple hot deck imputation and multivariate imputation by chained equations with two different model specifications: logistic regression and random forests. The methods are tested on real data from a questionnaire-based study in the Norwegian opioid maintenance treatment programme.Results:All methods performed relatively well when the sample size was large (n = 1000). For a smaller sample size (n = 200), the regression estimates depend heavily on the level of missing. When the amount of missing was ⩾20%, in particular, complete case analysis, hot deck and random forests had biased estimates with too low coverage. Multiple imputation using multiple correspondence analysis had the best performance all over.Conclusion:The choice of missing handling methodology has a significant impact on the clinical interpretation of the accompanying statistical analyses. With missing data, the choice of whether to impute or not, and choice of imputation method, can influence clinical conclusion drawn from a regression model and should therefore be given sufficient consideration.

Life expectancy with and without cognitive impairment among Chilean older adults: results of the National Survey of Health (2003, 2009 and 2016)

“…However, sex differences and time trends were similar to those observed in our primary analysis. The combination of MMSE and PFAQ scores which we used are a more accurate screening tool of CI for the Chilean older population than MMSE alone [19]. We found an increase in LE free of CI and a decrease in LE with CI between 2003 and 2016–17.…”

“…This instrument was derived from a previously validated version of the MMSE [18]. Quiroga et al [19] showed that a combination of MMSE and the Pfeffer Functional Activities Questionnaire (PFAQ) had a higher sensitivity and specificity to detect CI among the Chilean older population. Hence, the criteria used to determine if participants had CI were MMSE< 13 and PFAQ≥6.…”

Cross-trial prediction in psychotherapy: External validation of the Personalized Advantage Index using machine learning in two Dutch randomized trials comparing CBT versus IPT for depression

“…Besides the problem of small samples sizes, there is also a striking heterogeneity in statistical methods used for missing data imputation and model building (Cohen et al, 2019;. This is of importance since there is evidence that different methods of data imputation and model building may lead to differences in clinical conclusions (Cohen et al, 2019;Stavseth et al, 2019;Webb et al, 2020). However, comparisons between the STEPd PAI model built in 2015 (Huibers et al, 2015) and the PAI model in the present paper that used different methods on the same dataset, indicate significant overlap between selected pre-treatment variables and similar effect sizes.…”

Dissemination and implementation of thee-MCHHandbook, UNRWA’s newly released maternal and child health mobile application: a cross-sectional study

“…Model selection was performed with the minimisation of the Akaike information criteria in order to account for possible interactions among variables. In the regression analyses, missing data were imputed using random forests, which is a superior method of imputation for non-parametric variables and when there are many variables in the study 34…”