An assessment of statistical methods for nonindependent data in ecological meta‐analyses: Comment

Nakagawa, Shinichi; Senior, Alistair M.; Viechtbauer, Wolfgang; Noble, Daniel W. A.

doi:10.1002/ecy.3490

Cited by 16 publications

(18 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In such a case, averaging or sampling per species will not eliminate non-independence as effect sizes are still correlated via phylogeny (i.e. A in Equation 29; Nakagawa, Senior, et al, 2021). Furthermore, even when there is no phylogenetic signal ( 2 a = 0), or we do not have the species-level structure in a dataset, these alternative approaches could be problematic.…”

Section: Alternative Approaches: Averaging or Samplingmentioning

confidence: 99%

Methods for testing publication bias in ecological and evolutionary meta‐analyses

et al. 2021

Self Cite

View full text Add to dashboard Cite

1. Publication bias threatens the validity of quantitative evidence from meta-analyses as it results in some findings being overrepresented in meta-analytic datasets because they are published more frequently or sooner (e.g., 'positive' results). Unfortunately, methods to test for the presence of publication bias, or assess its impact on meta-analytic results, are unsuitable for datasets with high heterogeneity and non-independence, as is common in ecology and evolutionary biology.2. We first review both classic and emerging publication bias tests (e.g., funnel plots, Egger's regression, cumulative meta-analysis, fail-safe N, trim-and-fill tests, p-curve and selection models), showing that some tests cannot handle heterogeneity, and, more importantly, none of the methods can deal with non-independence. For each method we estimate current usage in ecology and evolutionary biology, based on a representative sample of 102 meta-analyses published in the last ten years.3. Then, we propose a new method using multilevel meta-regression, which can model both heterogeneity and non-independence, by extending existing regression-based methods (i.e.Egger's regression). We describe how our multilevel meta-regression can test not only publication bias, but also time-lag bias, and how it can be supplemented by residual funnel plots.4. Overall, we provide ecologists and evolutionary biologists with practical recommendations on which methods are appropriate to employ given independent and non-independent effect sizes.No method is ideal, and more simulation studies are required to understand how Type 1 and 2 error rates are impacted by complex data structures. Still, limitations of these methods do not justify ignoring publication bias in ecological and evolutionary meta-analyses.

show abstract

Section: Alternative Approaches: Averaging or Samplingmentioning

confidence: 99%

Methods for testing publication bias in ecological and evolutionary meta‐analyses

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…An approximate 95% Wald-type confidence interval for can be obtained with ̂ ± t .975,df SE[̂ ], where t .975,df denotes the 97.5th percentile of a t-distribution with df degrees of freedom. Based on Nakagawa et al (2021), we set df = N studies − 1, which we expected would bring the coverage rate of the confidence interval closer to its nominal 95% level (when compared to a confidence interval based on a standard normal distribution).…”

Section: Meta-analytic Modelsmentioning

confidence: 99%

“…Luke, 2017). Following Nakagawa et al (2021), we actually based the confidence interval on a t-distribution with N studies − 1 as the degrees of freedom (as an improvement to using a confidence interval based on a standard normal distribution), although this was apparently not conservative enough, presumably due to the additional dependency among the effect sizes introduced by the phylogeny. Further work will be needed to find an even better approximation to the degrees of freedom in the present context.…”

Section: Estimating the Overall Mean And Its Uncertaintymentioning

confidence: 99%

Phylogenetic multilevel meta‐analysis: A simulation study on the importance of modelling the phylogeny

2021

Self Cite

View full text Add to dashboard Cite

Meta‐analyses in ecology and evolution require special attention due to certain study characteristics in these fields. First, the primary articles in these fields usually report results that are observed from studies conducted with different species, and the phylogeny among the species violates the independence assumption. Second, articles frequently allow the computation of multiple effect sizes which cannot be accounted for by conventional meta‐analytic models. While both issues can be dealt with by utilizing a multilevel model that accounts for phylogeny, the performance of such a model has not been examined extensively. In this article, we investigate the performance of this model in comparison with some simpler models. We conducted an extensive simulation study where data with different hierarchical structures (in terms of study and species levels) were generated and then various models were fitted to examine their performance. The models we used include the conventional random effects and multilevel random‐effects models along with more complex multilevel models that account for species‐level variance with different variance components. Furthermore, we present an illustrative application of these models based on the data from a meta‐analysis on sizeassortative mating and comment on the results in light of the findings from the simulation study. Our simulation results show that, when the phylogenetic relationships among the species are at least moderately strong, only the most complex model that decomposes the species‐level variance into nonphylogenetic and phylogenetic components provides approximately unbiased estimates of the overall mean and variance components and yields confidence intervals with an approximately nominal coverage rate. Contrarily, removing the phylogenetic or non‐phylogenetic component leads to biased variance component estimates and an increased risk for incorrect inferences about the overall mean. These findings are supported by the results derived from the illustrative application. Based on our results, we suggest that meta‐analyses in ecology and evolution should use the model that accounts for both the nonphylogenetic and phylogenetic species‐level variance in addition to the multilevel structure of the data. Any attempts to simplify this model, such as using only the phylogenetic variance component, may lead to erroneous inferences from the data.

show abstract

“…For example, multiple effect sizes may be taken from a single study on traits measured on the same individuals or from many different species that share an evolutionary history (Cinar et al, 2021 preprint;Noble et al, 2017). As such, it is far more likely that comparative physiologists will need to apply a multilevel meta-analytic model to control for the various sources of non-independence (Cinar et al, 2021 preprint;Nakagawa and Santos, 2012;Nakagawa et al, 2021c;Noble et al, 2017;Song et al, 2021). Multilevel models are also extremely useful in describing the various drivers of effect size variance (a feature we describe below).…”

Section: Othermentioning

confidence: 99%

“…because traits are measured on the same sets of individuals), then we need to remove this dependency from the calculation of each v i ; we can then account for the assumed sampling covariance between effect sizes by modifying I to include off-diagonals that describe this covariance (see the Appendix and Noble et al, 2017). Alternatively, robust variance estimators can also be used to deal with effect size non-independence (Nakagawa et al, 2021c;Song et al, 2021). The fact that we can obtain an overall estimate of sampling variance (i.e.…”

Section: Othermentioning

confidence: 99%

Meta-analytic approaches and effect sizes to account for ‘nuisance heterogeneity’ in comparative physiology

Noble

Pottier

Lagisz

et al. 2022

Journal of Experimental Biology

Self Cite

View full text Add to dashboard Cite

Meta-analysis is a powerful tool used to generate quantitatively informed answers to pressing global challenges. By distilling data from broad sets of research designs and study systems into standardised effect sizes, meta-analyses provide physiologists with opportunities to estimate overall effect sizes and understand the drivers of effect variability. Despite this ambition, research designs in the field of comparative physiology can appear, at the outset, as being vastly different to each other because of ‘nuisance heterogeneity’ (e.g. different temperatures or treatment dosages used across studies). Methodological differences across studies have led many to believe that meta-analysis is an exercise in comparing ‘apples with oranges’. Here, we dispel this myth by showing how standardised effect sizes can be used in conjunction with multilevel meta-regression models to both account for the factors driving differences across studies and make them more comparable. We assess the prevalence of nuisance heterogeneity in the comparative physiology literature – showing it is common and often not accounted for in analyses. We then formalise effect size measures (e.g. the temperature coefficient, Q10) that provide comparative physiologists with a means to remove nuisance heterogeneity without the need to resort to more complex statistical models that may be harder to interpret. We also describe more general approaches that can be applied to a variety of different contexts to derive new effect sizes and sampling variances, opening up new possibilities for quantitative synthesis. By using effect sizes that account for components of effect heterogeneity, in combination with existing meta-analytic models, comparative physiologists can explore exciting new questions while making results from large-scale data sets more accessible, comparable and widely interpretable.

show abstract

An assessment of statistical methods for nonindependent data in ecological meta‐analyses: Comment

Cited by 16 publications

References 24 publications

Methods for testing publication bias in ecological and evolutionary meta‐analyses

Methods for testing publication bias in ecological and evolutionary meta‐analyses

Phylogenetic multilevel meta‐analysis: A simulation study on the importance of modelling the phylogeny

Meta-analytic approaches and effect sizes to account for ‘nuisance heterogeneity’ in comparative physiology

Contact Info

Product

Resources

About