Bootstrap inference when using multiple imputation

Abstract

Many modern estimators require bootstrapping to calculate confidence intervals because either no analytic standard error is available or the distribution of the parameter of interest is nonsymmetric. It remains however unclear how to obtain valid bootstrap inference when dealing with multiple imputation to address missing data. We present 4 methods that are intuitively appealing, easy to implement, and combine bootstrap estimation with multiple imputation. We show that 3 of the 4 approaches yield valid inference, but that the performance of the methods varies with respect to the number of imputed data sets and the extent of missingness. Simulation studies reveal the behavior of our approaches in finite samples. A topical analysis from HIV treatment research, which determines the optimal timing of antiretroviral treatment initiation in young children, demonstrates the practical implications of the 4 methods in a sophisticated and realistic setting. This analysis suffers from missing data and uses the g-formula for inference, a method for which no standard errors are available.

Cost-effectiveness analysis of a randomized clinical trial of early versus deferred endovenous ablation of superficial venous reflux in patients with venous ulceration

“…The base‐case analysis used bootstrapping, with 1000 Monte Carlo resamples with replacement. The bootstrap was used only for the analysis of complete cases, as bootstrap combined with multiple imputation can be very complex. As an alternative method in sensitivity analyses, standard errors and correlation between total costs and QALYs were estimated assuming bivariable normality ( Appendix S1, supporting information).…”

Cost‐utility of a six‐month programmed sports therapy (PST) in patients with haemophilia

“…For imputed data, the overall difference in mean costs and QALYs between treatment groups was calculated using Rubin's rules as the average of estimates from each of the multiple imputed data sets . The calculation of appropriate uncertainty intervals followed the approach of Schomaker and Heumann . Therefore, we bootstrapped the data including missing values in a first step (4000 replications).…”

“…28 The calculation of appropriate uncertainty intervals followed the approach of Schomaker and Heumann. 29 (Figures 1 and 3). The CEACs describe the probability that the intervention is cost-effective in comparison with care as usual for a range of ceiling ratios, which are defined as willingness to pay to gain one unit of effect (here QALY) ( Figure 2).…”

Bootstrapping and multiple imputation ensemble approaches for classification problems

“…In our case, we are using small value of MI, which means there will be higher uncertainty in the estimates. As commented by Schomaker and Heumann [30], bootstrapping of MI may be preferred for smaller imputation uncertainty (or moderate to large values of M). Therefore, we do not use this type of ensemble technique in this paper.…”

“…However, if the missingness is moderate to large, then MI followed by tree-bagging is useful. Schomaker and Heumann [30] comment that MI on bootstrapped samples and bootstrapped samples on multiple imputed datasets are the best options to calculate randomization valid confidence intervals when combining bootstrapping with MI. They further suggest that MI of bootstrap samples may be preferred for large imputation uncertainty (or low missingness) and bootstrapping of MI may be preferred for smaller imputation uncertainty (high missingness).…”