A Bayesian Perspective on the Reproducibility Project: Psychology

Etz, Alexander; Vandekerckhove, Joachim

doi:10.1371/journal.pone.0149794

Cited by 281 publications

(267 citation statements)

References 29 publications

Supporting

Mentioning

250

Contrasting

Unclassified

Order By: Relevance

“…57-59). According to a Bayesian reanalysis of the Reproducibility Project, one pair of authors argued that "the apparent discrepancy between the original set of results and the outcome of the Reproducibility Project can be explained adequately by the combination of deleterious publication practices and weak standards of evidence, without recourse to hypothetical hidden moderators" (60). However, this paper did not directly code or analyze contextual sensitivity in any systematic way.…”

mentioning

confidence: 97%

Contextual sensitivity in scientific reproducibility

Bavel

Mende-Siedlecki

Brady

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

364

313

View full text Add to dashboard Cite

In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.replication | reproducibility | context | psychology | meta-science I n recent years, scientists have paid increasing attention to reproducibility. Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6). Indeed, since the launch of the clinicaltrials.gov registry in 2000, which forced researchers to preregister their methods and outcome measures, the percentage of large heart-disease clinical trials reporting significant positive results plummeted from 57% to a mere 8% (7). The costs of such irreproducible preclinical research, estimated at $28 billion in the United States (8), are staggering. In a similar vein, psychologists have expressed growing concern regarding the reproducibility and validity of psychological research (e.g., refs. 9-14). This emphasis on reproducibility has produced a number of failures to replicate prominent studies, leading professional societies and government funding agencies such as the National Science Foundation to form subcommittees promoting more robust research practices (15).The Reproducibility Project in psychology has b...

show abstract

mentioning

confidence: 97%

Contextual sensitivity in scientific reproducibility

Bavel

Mende-Siedlecki

Brady

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

364

313

View full text Add to dashboard Cite

show abstract

“…On top of that, many available tools for professionals and students are either overpriced, too complex (i.e., displaying vast amounts of raw information neither demanded nor needed by the user) or too basic (i.e., not supporting advanced statistical procedures). These factors contribute to the reproducibility crisis in psychological science (Chambers et al 2014, Etz andVandekerckhove (2016), Szucs and Ioannidis (2016)). …”

mentioning

confidence: 99%

The psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science

Makowski¹

2018

JOSS

325

201

View full text Add to dashboard Cite

SummaryStatistics are a vital aspect of psychological science. Unfortunately, the process of analyzing, formatting and reporting results is often fastidious, time-consuming and prone to errors, resulting in frustration and aversion. On top of that, many available tools for professionals and students are either overpriced, too complex (i.e., displaying vast amounts of raw information neither demanded nor needed by the user) or too basic (i.e., not supporting advanced statistical procedures). These factors contribute to the reproducibility crisis in psychological science (Chambers et al. 2014, Etz andVandekerckhove (2016), Szucs and Ioannidis (2016)).Psycho is an R package (Team 2000) whose main goal is to fill the gap between statistical analyses and publication-ready documents, involving information selection, results transformation and text formatting. These steps, beyond being the most time-consuming part of statistical analysis in psychological science, are the source of many human errors and can lead to a badly formatted results section, veiling the general quality of a paper. The psycho package, built for psychologists, neuropsychologists and neuroscientists, also implements commonly used routines such as correlation matrices, assessment plot creation or standardization. The package revolves around the psychobject, returned by most of its functions. Additionally, the analyze() function transforms other R objects into psychobjects. Four methods can then be applied to them: summary(), print(), plot() and values(), resulting in a simple yet flexible workflow.A special focus has been dedicated to the support of cutting-edge statistical routines, such as mixed-models. Moreover, a specific attention has been devoted to the promotion, through support and documentation, of the Bayesian framework. Indeed, beyond the many methodological reasons to prefer this approach (better accuracy in noisy data, the possibility of introducing prior knowledge into the analysis and, critically, results intuitiveness and their straightforward interpretation; (Lee 2012, Gabry and Goodrich (2016))), its core ideology (computing the probability of different effects compatible with the observed data) sets it as the right way of answering the questions of psychology.Contrary to many other packages, whose goal is to produce statistical analyses, psycho bridges statistical R outputs with statistical reports writing, with a focus on APA formatting guidelines, to enhance the standardization of results reporting. Complex outputs, such as those of Bayesian and frequentist mixed models, are automatically transformed into readable text, tables, and plots that illustrate the effects. With this package, results can be easily incorporated into shareable reports and publications, promoting data exploration, saving time and preventing errors for better, reproducible, science.The psycho package can be downloaded and installed from CRAN 1. Source code is available from its GitHub repository 2. Documentation 3, examples 4 and issues tracker 5 are also p...

show abstract

“…If H 0 :H 1 odds = 6 then FRP will be 67.92%. Looking at these numbers the replication crisis does not seem surprising: using NHST very high FRP can be expected even with modestly high H 0 :H 1 odds and moderate bias (Etz and Vandekerckhove, 2016). Hence, under realistic conditions FRP not only extremely rarely equals α or the p-value (and TRP extremely rarely equals 1-α and/or 1-p-value) but also, FRP is much larger than the generally assumed 5% and TRP is much lower than the generally assumed 95%.…”

mentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.

show abstract

A Bayesian Perspective on the Reproducibility Project: Psychology

Cited by 281 publications

References 29 publications

Contextual sensitivity in scientific reproducibility

Contextual sensitivity in scientific reproducibility

The psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science

When null hypothesis significance testing is unsuitable for research: a reassessment

Contact Info

Product

Resources

About