Investigating Variation in Replicability

Klein, Richard Anthony; Ratliff, Kate A.; Vianello, Michelangelo; Adams, Reginald B.; Bahník, Štěpán; Bernstein, Michael J.; Bocian, Konrad; Brandt, Mark J.; Brooks, Beach Smith; Brumbaugh, Claudia Chloe; Cemalcılar, Zeynep; Chandler, Jesse; Cheong, Winnee; Davis, William E.; Devos, Thierry; Eisner, Matthew; Frankowska, Natalia; Furrow, David; Galliani, Elisa Maria; Hasselman, Fred; Hicks, Joshua A.; Hovermale, James F.; Hunt, S. Jane; Huntsinger, Jeffrey R.; IJzerman, Hans; John, Melissa Sue; Joy-Gaba, Jennifer A.; Kappes, Heather Barry; Krueger, Lacy E.; Kurtz, Jaime L.; Levitan, Carmel; Mallett, Robyn K.; Morris, Wendy L.; Nelson, Anthony J.; Nier, Jason A.; Packard, Grant; Pilati, Ronaldo; Rutchick, Abraham M.; Schmidt, Kathleen; Skorinko, Jeanine; Smith, Robert J.; Steiner, Troy G.; Storbeck, Justin; Swol, Lyn M. Van; Thompson, Donna; Veer, Anna Elisabeth van 't; Vaughn, Leigh Ann; Vranka, Marek; Wichman, Aaron L.; Woodzicka, Julie A.; Nosek, Brian A.

doi:10.1027/1864-9335/a000178

Cited by 988 publications

(691 citation statements)

References 37 publications

Supporting

Mentioning

617

Contrasting

Unclassified

Order By: Relevance

“…For instance, context was expected to be largely irrelevant for research on visual statistical learning (rated 1) (68) or for the action-based model of cognitive dissonance (rated 2) (69), was expected to have some influence on research concerning bilingualism and inhibitory control (rated 3) (70), and was expected to have a significant impact on research on the ultimate sampling dilemma (rated 4) (71) and on whether cues regarding diversity signal threat or safety for African Americans (rated 5) (72). † It is worth noting that Many Labs 1 (61) found considerable heterogeneity of effect sizes for nearly half of their effects (6/13). Furthermore, they found sample (United States vs. international) and setting (online vs. in-lab) differences for nearly one-third (10/32) of their moderation tests, seven of which were among the largest effects (i.e., anchoring, allowed-forbidden).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Contextual sensitivity in scientific reproducibility

Bavel

Mende-Siedlecki

Brady

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

362

313

View full text Add to dashboard Cite

In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.replication | reproducibility | context | psychology | meta-science I n recent years, scientists have paid increasing attention to reproducibility. Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6). Indeed, since the launch of the clinicaltrials.gov registry in 2000, which forced researchers to preregister their methods and outcome measures, the percentage of large heart-disease clinical trials reporting significant positive results plummeted from 57% to a mere 8% (7). The costs of such irreproducible preclinical research, estimated at $28 billion in the United States (8), are staggering. In a similar vein, psychologists have expressed growing concern regarding the reproducibility and validity of psychological research (e.g., refs. 9-14). This emphasis on reproducibility has produced a number of failures to replicate prominent studies, leading professional societies and government funding agencies such as the National Science Foundation to form subcommittees promoting more robust research practices (15).The Reproducibility Project in psychology has b...

show abstract

Section: Methodsmentioning

confidence: 99%

“…One large-scale replication tested 13 effects (10 were reproduced consistently, and one was reproduced weakly) across 36 international samples (61). They observed only small effects of setting and a much stronger influence of the effects themselves (i.e., some effects are simply more robust than others, regardless of setting).…”

mentioning

confidence: 99%

Contextual sensitivity in scientific reproducibility

Bavel

Mende-Siedlecki

Brady

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

362

313

View full text Add to dashboard Cite

show abstract

“…The results of such attempts have varied greatly. In psychology, where Ioannidis' arguments can be expected to hold as well, the journal Social Psychology published a special issue that reported replications of 13 recent studies (Klein et al 2015). In 10 out of 13 cases, the effects reported in the original papers were found again, although often with a smaller magnitude.…”

Section: Introductionmentioning

confidence: 92%

Randomness and the Games of Science

Goeman

2016

The Frontiers Collection

View full text Add to dashboard Cite

Recently it has become clear that too many findings reported in the scientific literature are irreproducible. We study the causes of this phenomenon from a statistical perspective. Although a certain amount of irreproducible research is unavoidable due to the randomness inherent to scientific observation, two related phenomena conspire to increase the proportion of such findings: publication bias, i.e. the custom that negative findings are usually not published, and confirmation bias, i.e. the human inclination to interpret observations in a way that confirms prior beliefs. Both biases are poorly held in check in the current scientific publication model in which there is no explicit role for the views of a critic, i.e. a scientist with opposing theoretical views. We argue that if researchers are able to play the critic's role imaginatively, they will publish science of higher methodological quality that is not only more reproducible, but also more relevant for theory. To allow for this, we must promote a different view on statistical methodology, seeing statistics not as the gatekeeper of scientific evidence, but as a language scientists may use to discuss uncertainty when they talk about the implications of observations for theory.

show abstract

“…Besides in psychology, reproducibility problems have also been indicated in economics (Camerer, et al, 2016) and medicine (Begley, & Ellis, 2012). Although these studies suggest substantial evidence of false positives in these fields, replications show considerable variability in resulting effect size estimates (Klein, et al, 2014;Stanley, & Spence, 2014). Therefore caution is warranted when wishing to draw conclusions on the presence of an effect in individual studies (original or replication;Open Science Collaboration, 2015;Gilbert, King, Pettigrew, & Wilson, 2016;Anderson, et al 2016).…”

mentioning

confidence: 99%

Too Good to be False: Nonsignificant Results Revisited

Hartgerink

Wicherts

Assen

2017

Collabra: Psychology

View full text Add to dashboard Cite

Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This might be unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We examined evidence for false negatives in nonsignificant results in three different ways. We adapted the Fisher test to detect the presence of at least one false negative in a set of statistically nonsignificant results. Simulations show that the adapted Fisher method generally is a powerful method to detect false negatives. We examined evidence for false negatives in the psychology literature in three applications of the adapted Fisher method. These applications indicate that (i) the observed effect size distribution of nonsignificant effects exceeds the expected distribution assuming a null-effect, and approximately two out of three (66.7%) psychology articles reporting nonsignificant results contain evidence for at least one false negative, (ii) nonsignificant results on gender effects contain evidence of true nonzero effects, and (iii) the statistically nonsignificant replications from the Reproducibility Project Psychology (RPP) do not warrant strong conclusions about the absence or presence of true zero effects underlying these nonsignificant results. We conclude that false negatives deserve more attention in the current debate on statistical practices in psychology. Potentially neglecting effects due to a lack of statistical power can lead to a waste of research resources and stifle the scientific discovery process.

show abstract

Investigating Variation in Replicability

Cited by 988 publications

References 37 publications

Contextual sensitivity in scientific reproducibility

Contextual sensitivity in scientific reproducibility

Randomness and the Games of Science

Too Good to be False: Nonsignificant Results Revisited

Contact Info

Product

Resources

About