Comparing traditional and Bayesian approaches to ecological meta‐analysis

Abstract

1. Despite the wide application of meta-analysis in ecology, some of the traditional methods used for meta-analysis may not perform well given the type of data characteristic of ecological meta-analyses. 2. We reviewed published meta-analyses on the ecological impacts of global climate change, evaluating the number of replicates used in the primary studies (n i) and the number of studies or records (k) that were aggregated to calculate a mean effect size. We used the results of the review in a simulation experiment to assess the performance of conventional frequentist and Bayesian meta-analysis methods for estimating a mean effect size and its uncertainty interval. 3. Our literature review showed that n i and k were highly variable, distributions were right-skewed and were generally small (median n i = 5, median k = 44). Our simulations show that the choice of method for calculating uncertainty intervals was critical for obtaining appropriate coverage (close to the nominal value of 0.95). When k was low (<40), 95% coverage was achieved by a confidence interval (CI) based on the t distribution that uses an adjusted standard error (the Hartung-Knapp-Sidik-Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap or z distribution CIs had lower coverage. Despite the importance of the method to calculate the uncertainty interval, 39% of the meta-analyses reviewed did not report the method used, and of the 61% that did, 94% used a potentially problematic method, which may be a consequence of software defaults. 4. In general, for a simple random-effects meta-analysis, the performance of the best frequentist and Bayesian methods was similar for the same combinations of factors (k and mean replication), though the Bayesian approach had higher than nominal (>95%) coverage for the mean effect when k was very low (k < 15). Our literature review suggests that many meta-analyses that used z distribution or bootstrapping CIs may have overestimated the statistical significance of their results when the number of studies was low; more appropriate methods need to be adopted in ecological meta-analyses.

Phylogenetic multilevel meta‐analysis: A simulation study on the importance of modelling the phylogeny

“…The present results therefore reflect the performance of the various models under idealized conditions (i.e., when the sample sizes of the individual studies are sufficiently large, such that the sampling distributions of the estimates are indeed approximately normal and any inaccuracies in the estimated sampling variances are negligible). Although such ideal conditions are rare in practice (Hillebrand and J. Gurevitch, 2014;Pappalardo et al, 2020), the advantage of using a generic measure is that we were able to identify problems that are inherent to certain models and not (potentially) a consequence of violations to the model assumptions (i.e., if a particular model performs poorly for a measure that violates model assumptions, we do not know whether the poor performance is attributable to deficiencies of the model itself or a consequence of model assumptions being violated). On the other hand, it remains to be determined how well the phylogenetic multilevel model performs when the effect sizes are generated based on the exact distributional assumptions underlying specific measures.…”

An assessment of statistical methods for nonindependent data in ecological meta‐analyses: Comment

“…This involved simply using one less than the total number of papers instead of the typical degrees of freedom that uses the total number of effect sizes (i.e., df = total papers -1); 2) a Satterthwaite approximation to the effective degrees of freedom, which is commonly applied in the linear mixed effect model literature (Satterthwaite, 1946); 3) a second cluster-robust estimation method implemented in the clubSandwich package in R (Pustejovsky, 2020) that uses a bias-reduced linearization method (Pustejovsky and Tipton, 2018). The R package clubSandwich uses a similar robust-variance estimation method as robumeta (Fisher et al 2017) used in Song et al (2020), but can be applied to metafor's rma.mv model objects; and 4) a Bayesian modelling approach that uses an MCMC algorithm (using the R package MCMCglmm -Hadfield, 2010), instead of restricted maximum likelihood (REML) estimation, as MCMC algorithms are known to have robust coverage, albeit are slightly conservative with small sample sizes (Pappalardo et al 2020). We also explored other modelling approaches, but present these four as they are simple solutions that can be easily implemented.…”

Bayesian Methods for Meta-Analyses of Binary Outcomes: Implementations, Examples, and Impact of Priors

“…Instead, we aimed at providing some practical guidelines for researchers who are interested in implementing Bayesian meta-analyses. One may refer to other papers that especially aimed at comparing these two types of methods to better understand the pros and cons of frequentist and Bayesian meta-analyses [ 74 , 79 , 80 , 81 , 82 , 83 , 84 ].…”