Heterogeneity in ecological and evolutionary meta‐analyses: its magnitude and implications

Abstract

Meta-analysis is the gold standard for synthesis in ecology and evolution. Together with estimating overall effect magnitudes, meta-analyses estimate differences between effect sizes via heterogeneity statistics. It is widely hypothesized that heterogeneity will be present in ecological/evolutionary meta-analyses due to the system-specific nature of biological phenomena. Despite driving recommended best practices, the generality of heterogeneity in ecological data has never been systematically reviewed. We reviewed 700 studies, finding 325 that used formal meta-analysis, of which total heterogeneity was reported in fewer than 40%. We used second-order meta-analysis to collate heterogeneity statistics from 86 studies. Our analysis revealed that the median and mean heterogeneity, expressed as I , are 84.67% and 91.69%, respectively. These estimates are well above "high" heterogeneity (i.e., 75%), based on widely adopted benchmarks. We encourage reporting heterogeneity in the forms of I and the estimated variance components (e.g., τ ) as standard practice. These statistics provide vital insights in to the degree to which effect sizes vary, and provide the statistical support for the exploration of predictors of effect-size magnitude. Along with standard meta-regression techniques that fit moderator variables, multi-level models now allow partitioning of heterogeneity among correlated (e.g., phylogenetic) structures that exist within data.

Comparing the Ecological Stoichiometry in Green and Brown Food Webs – A Review and Meta-analysis of Freshwater Food Webs

“…While a temperature scaling coefficient could standardize metabolic rates across varying temperatures (e.g., Vanni and Mcintyre, 2016), such standardization would not affect our inferences because each effect size was calculated from individuals held at the same temperature. Many of the factors that differed across studies likely drove high heterogeneity (I 2 ) across effect sizes ( Table 1 ), but this heterogeneity was accounted by using a mixed effects model and I 2 was similar to that reported across other meta-analyses in ecology (Senior et al, 2016). Finally, our classification of benthic invertebrates into herbivores versus detritivores was based solely on diets fed in experiments, and may not reflect feeding ecology or the stoichiometry of feeding in the field, where animals can feed selectively on nutrient-rich biofilms (Hood et al, 2014) or forage on multiple resource types and confound trophic classification (Wolkovich et al, 2014; Snyder et al, 2015; Stoler et al, 2016).…”

“…The use of random effects accounts for heterogeneity across studies due to variable factors including temperature, taxonomy, and diet. We assessed heterogeneity of effect sizes across studies using the I 2 statistic, which equates to the proportion of total heterogeneity attributable to between-study variance ( Table 1 ) (Senior et al, 2016). Because of insufficient datasets regarding N and P egestion by herbivores, we decided to exclude herbivores from the meta-analysis of those effect sizes and focus only on detritivore datasets.…”

Critically examining the knowledge base required to mechanistically project climate impacts: A case study of Europe's fish and shellfish

“…Following Higgins, Thompson, Deeks, and Altman (), the I ² measurements (Supporting information Table S3) suggested a high degree of heterogeneity within the models. A similarly large amount of heterogeneity is considered normal in ecological meta‐analyses (Senior et al., ). More remarkable, however, was the contribution of the variance at specific levels to the overall heterogeneity.…”

Nonindependence and sensitivity analyses in ecological and evolutionary meta‐analyses

“…Senior et al . ). Thus, using a composite effect size could introduce a new set of problems (see Marín‐Martínez & Sánchez‐Meca ; Gleser & Olkin ).…”

“…It is notable, however, that Senior et al . () recently showed total heterogeneity in ecological and evolutionary meta‐analyses to be very high on average (~92%), which indicates that random‐effects models are more appropriate for typical meta‐analyses in ecology and evolution. Nonetheless, we should be aware that incorrect calculations of sampling error variance can lead to the incorrect estimation of heterogeneity in a meta‐analysis (i.e.…”