Heterogeneity in direct replications in psychology and its association with effect size.

Abstract

We examined the evidence for heterogeneity (of effect sizes) when only minor changes to sample population and settings were made between studies and explored the association between heterogeneity and average effect size in a sample of 68 meta-analyses from thirteen pre-registered multi-lab direct replication projects in social and cognitive psychology. Amongst the many examined effects, examples include the Stroop effect, the "verbal overshadowing" effect, and various priming effects such as "anchoring" effects. We found limited heterogeneity; 48/68 (71%) meta-analyses had non-significant heterogeneity, and most (49/68; 72%) were most likely to have zero to small heterogeneity. Power to detect small heterogeneity (as defined by Higgins, 2003) was low for all projects (mean 43%), but good to excellent for medium and large heterogeneity. Our findings thus show little evidence of widespread heterogeneity in direct replication studies in social and cognitive psychology, suggesting that minor changes in sample population and settings are unlikely to affect research outcomes in these fields of psychology. We also found strong correlations between observed average effect sizes (standardized mean differences and log odds ratios) and heterogeneity in our sample. Our results suggest that heterogeneity and moderation of effects is unlikely for a zero average true effect size, but increasingly likely for larger average true effect size.

Replicability, Robustness, and Reproducibility in Psychological Science

“…Further, there are multiple examples of presumed context sensitivity failing to occur or account for replication failures when examined directly (Ebersole, Atherton, et al, 2016;Ebersole et al, 2020;Klein et al, 2014Klein et al, , 2018. Heterogeneity is sometimes observed in replication studies, but it is usually modest and insufficient to make a replicable phenomenon appear or disappear based on factors that would not have been anticipated in advance of conducting the studies (Baribault et al, 2018;Klein et al, 2014Klein et al, , 2018Olsson-Collentine et al, 2020). Identifying circumstances in which the replicability of a finding is demonstrated to be contingent on unconsidered factors in the operationalization will be productive for advancing investigations of correlates of replicability.…”

Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias

“…Note that the mode of our default prior indicates homogeneity (τ 2 =0). We argue this is a sensible choice as we see within replication that when the true effect size is 0, there is often also no heterogeneity (Olsson-Collentine, et al, 2020). Given our default prior of ρ ~ ℕ(0,1), choosing no heterogeneity is a logical prior on heterogeneity.…”

“…However, a uniform prior may be neither desirable nor realistic. For instance, on the basis of previous empirical evidence researchers may expect a zero true effect size and no effect size heterogeneity in a large replication project (e.g., based on the many labs projects, Klein et al, 2014;Olsson-Collentine, Wicherts, & van Assen, 2020). Alternatively, a meta-analyst may have used very broad inclusion criteria for the studies, with a wide range of populations, treatments, and measurements of both independent and dependent variables as is mainstream in meta-analyses in psychology, and therefore expecting large heterogeneity of effect size (van Erp, Verhagen, Grasman, & Wagenmakers, 2017).…”

The case for default point-H1-hypotheses in behavioral science: a theory-construction perspective

“…Furthermore, several recent meta-meta-analyses of behavioral science findings show that the homogenous partition of observed effects is typically comprised of small effects (0.01≤d≤0.20), whereas the relatively infrequent medium to large effects (0.50≤d≤1.00) tend to be highly heterogeneous (Olsson-Collentine, Wicherts, & van Assen, 2020;Schauer & Hedges, 2020;Linden & Hönekopp, 2020). We already saw that replication studies that observe a large(r) heterogeneous effect are theoretically ambiguous, because such effects require a differentiated explanation of the observed heterogeneity (which is hard to come by); whereas replication studies that observe a small homogeneous effect are theoretically uninteresting, because the observed effect is overlain by the measurement-error.…”