Contextual sensitivity in scientific reproducibility

Abstract

In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.replication | reproducibility | context | psychology | meta-science I n recent years, scientists have paid increasing attention to reproducibility. Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6). Indeed, since the launch of the clinicaltrials.gov registry in 2000, which forced researchers to preregister their methods and outcome measures, the percentage of large heart-disease clinical trials reporting significant positive results plummeted from 57% to a mere 8% (7). The costs of such irreproducible preclinical research, estimated at $28 billion in the United States (8), are staggering. In a similar vein, psychologists have expressed growing concern regarding the reproducibility and validity of psychological research (e.g., refs. 9-14). This emphasis on reproducibility has produced a number of failures to replicate prominent studies, leading professional societies and government funding agencies such as the National Science Foundation to form subcommittees promoting more robust research practices (15).The Reproducibility Project in psychology has b...

Spawning behavior of Arctic charr (Salvelinus alpinus): Spawning synchrony, vibrational communication, and mate guarding

“…Although replicating previous observations is in itself relevant (Ioannidis, 2005; Van Bavel, Mende‐Siedlecki, Brady, & Reinero, 2016), the last decade's technological development of cameras and video quality additionally enables us to operate more cameras and hence record more spawning events at video resolutions revealing behaviors previously not documented in our population (e.g., egg eating including filial cannibalism). Moreover, the vibrations of charr during courtship and spawning leave a recordable sound track in the water column (discovered here by MBB).…”

Making Replication Mainstream

“…A headline finding from this project was that only 36% of the attempted replications were "successful," in the sense that a significant effect in the same direction as the original was found. The publication of this report was met with a wide range of responses, including some focused on the fidelity of the replication studies, the criteria used to determine whether a replication was successful, the value of such a large-scale investment of resources, and so on (e.g., Anderson et al, 2016;Etz & Vandekerckhove, 2016;Gilbert, King, Pettigrew, 2016;Kunert, 2016;Maxwell, Lau, & Howard, 2015;Morey & Lakens, 2016;van Aert & van Assen, 2017;Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016). These responses (which continue to be published at the time of our writing this paper) focus on a broad range of challenges and objections to the value of replication studies as a whole.…”

Meta-assessment of bias in science

“…However, it is yet unknown to what extent bias patterns and postulated risk factors are generalizable phenomena that threaten all scientific fields in similar ways and whether studies documenting such problems are reproducible (5)(6)(7). Indeed, evidence suggests that biases may be heterogeneously distributed in the literature.…”