Using bootstrapping to account for sampling variation and large study bias in meta-analysis

Abstract

Meta-analysis is a commonly used tool to provide insight into effect estimates with high confidence due to the increase in power and resistance to sampling bias compared to individual studies. However, meta-analysis is not without flaws as bias from original research can easily carry over to the synthesized analysis. In the present study, we used non-parametric bootstrapping to address the issues of non-random sampling (i.e., the fact that meta-analyses aim to include all available data instead of random samples from the population), as well as large study bias in meta-analysis. We found that bootstrapping can be highly informative regarding (1) the robustness of effect estimates despite non-random sampling, (2) how the inclusion of additional datasets affects the effect estimate and its error, and (3) to identify influential datasets. We furthermore found that the standard deviation of the effect estimate decreases logarithmically with the addition of further datasets. Regarding large study bias, we found that (4) even in random-effects models, very large datasets have the potential to bias the data significantly. Using a bias adjustment method that replaces sample sizes of individual datasets by their square root radicand, we could account for the bias without losing a significant amount of information. Overall, we suggest that bootstrapping should be routinely used to estimate the robustness of effect estimates and assess large study bias in meta-analysis.