Recommendations for Increasing Replicability in Psychology

Asendorpf, Jens B.; Conner, Mark; Fruyt, Filip De; Houwer, Jan De; Denissen, Jaap J. A.; Fiedler, Klaus; Fiedler, Susann; Funder, David C.; Kliegl, Reinhold; Nosek, Brian A.; Perugini, Marco; Roberts, Brent W.; Schmitt, Manfred; Aken, Marcel A. G. van; Weber, Hannelore; Wicherts, Jelte M.

doi:10.1002/per.1919

Cited by 752 publications

(615 citation statements)

References 48 publications

Supporting

Mentioning

570

Contrasting

Unclassified

Order By: Relevance

“…Transparency of methods and data is a core value of science [1][2][3][4] and is presumed to help increase the reproducibility of scientific evidence [5][6]. However, sharing of research materials, data, and supporting analytic code is the exception rather than the rule [7].…”

Section: Introductionmentioning

confidence: 99%

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

Kidwell¹,

Lazarević²,

Baranski³

et al. 2016

Preprint

Self Cite

135

167

View full text Add to dashboard Cite

Data Availability Statement: All data and materials are publicly accessible at https://osf.io/rfgdw/. Also, we preregistered our study design and analysis plan. You can find the preregistration at https://osf.io/ipkea/.Funding: The authors received no specific funding for this work.Competing Interests: Brian Nosek created the badges to acknowledge open practices, and Brian Nosek and Mallory Kidwell are on a committee maintaining the badges. The badges and specifications for earning them are CC0 licensed with and in four comparison journals. We report an increase in reported data sharing of more than an order of magnitude from baseline in Psychological Science, as well as an increase in reported materials sharing, although to a weaker degree. Moreover, we show that reportedly available data and materials were more accessible, correct, usable, and complete when badges were earned. We demonstrate that badges are effective incentives that improve the openness, accessibility, and persistence of data and materials that underlie scientific research.

show abstract

Section: Introductionmentioning

confidence: 99%

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

Kidwell¹,

Lazarević²,

Baranski³

et al. 2016

Preprint

Self Cite

135

167

View full text Add to dashboard Cite

show abstract

“…One problem with this procedure is that it can be difficult to demonstrate a failure to replicate when the original experiment was so low in power that 1 See http://www.openscienceframework.org/project/EZcUj/wiki/home. the confidence interval on effect size is relatively broad and almost covers zero (Asendorpf et al, 2013).…”

Section: Introductionmentioning

confidence: 94%

Bayesian tests to quantify the result of a replication attempt.

Verhagen¹,

Wagenmakers²

2014

Journal of Experimental Psychology: General

280

357

View full text Add to dashboard Cite

Replication attempts are essential to the empirical sciences. Successful replication attempts increase researchers' confidence in the presence of an effect, whereas failed replication attempts induce skepticism and doubt. However, it is often unclear to what extent a replication attempt results in success or failure. To quantify replication outcomes we propose a novel Bayesian replication test that compares the adequacy of two competing hypotheses. The first hypothesis is that of the skeptic and holds that the effect is spurious; this is the null hypothesis that postulates a zero effect size, H 0 : δ = 0. The second hypothesis is that of the proponent and holds that the effect is consistent with the one found in the original study, an effect that can be quantified by a posterior distribution. Hence, the second hypothesis -the replication hypothesis-is given by H r : δ ∼ "posterior distribution from original study". The weighted likelihood ratio between H 0 and H r quantifies the evidence that the data provide for replication success and failure. In addition to the new test, we present several other Bayesian tests that address different but related questions concerning a replication study. These tests pertain to the independent conclusions of the separate experiments, the difference in effect size between the original experiment and the replication attempt, and the overall conclusion based on the pooled results. Together, this suite of Bayesian tests allows a relatively complete formalization of the way in which the result of a replication attempt alters our knowledge of the phenomenon at hand. The use of all Bayesian replication tests is illustrated with three examples from the literature. For experiments analyzed using the t test, computation of the new replication test only requires the t values and the numbers of participants from the original study and the replication study.

show abstract

“…Gemäß Asendorpf et al (2013) (Bacon, 1267(Bacon, /1859Jasny, Chin, Chong, & Vignieri, 2011;Kuhn, 1962;Popper, 1934Popper, /1992Rosenthal, 1991) und sogar als Demarkationslinie zwischen Wissenschaft und Nicht-Wissenschaft bezeichnet wird, (Braude, 1979), wurde schon seit längerem in Frage gestellt, ob die gängige Wissenschaftspraxis tatsächlich zu in der Regel replizierbaren Befunden führt 1 (Cohen, 1994, Ionannidis, 2005Makel, Plucker, & Hegarty, 2012;Maxwell, 2004;Meehl, 1990). In jüngerer Zeit gab es insbesondere im Fach Psychologie (Ebersole, Atherton, Belanger, Skulborstad, Adams, Allen et al, 2014;Klein et al, 2014;Klein et al, 2016;Open Science Collaboration, 2015), aber 1 Selbstverständlich sind dennoch Studien denkbar, die sich nicht (mehr) replizieren lassen, obwohl die Originalstudie einen "wahren" Befund berichtete, beispielsweise weil es keine weitere Beobachtungsgelegenheit gibt oder weil der wahre Effekt nach gesellschaftlichen Veränderungen nicht mehr existiert (für eine ausführlichere Darstellung von Replizierbarkeit und ihrer Quantifizierung siehe Asendorpf et al, 2013oder auch Valentine et al, 2011oder aber Cumming und Finch, 2005.…”

Section: Wege Aus Der Vertrauenskriseindividuelle Schritte Hin Zu Verunclassified

Wege aus der Vertrauenskrise – Individuelle Schritte hin zu verlässlicher und offener Forschung

Geukes¹,

Schönbrodt²,

Utesch³

et al. 2016

Preprint

View full text Add to dashboard Cite

Die Psychologie, und mit ihr auch die Sportpsychologie, befindet sich in einer sogenannten Vertrauenskrise (siehe Titel des Themenhefts). Während die Diskussion ihrer Ursachen und Konsequenzen lebendig geführt wird, fehlt es an der Herausarbeitung individueller Möglichkeiten, Vertrauen wieder kollektiv herzustellen. In diesem Beitrag zeigen wir Wege aus dieser Vertrauenskrise auf und beschreiben insbesondere individuelle Schritte hin zu verlässlichem und offenem Forschen: Verlässliches Forschen bezieht sich auf das Publizieren von robusten Ergebnissen, sowie von direkten und konzeptuellen Replikationsstudien und offenes Forschen auf die Transparenz hinsichtlich der Planung (Präregistrierung), der Durchführung (Open Material) und der Analyse (Open Data, Reproducible Code) wissenschaftlicher Untersuchungen. Das Bekenntnis zu verlässlicher und offener Forschung wird unser Handeln in den verschiedenen Rollen im Wissenschaftssystem (als Forscher, Reviewer, Herausgeber, Betreuer, Kommissionsmitglied etc.) verändern. In diesem Sinne begreifen wir in diesem Beitrag die momentane Diskussion als Chance, die Zuverlässigkeit unserer Befunde nachhaltig zu steigern und langfristig gerechtfertigtes Vertrauen zu schaffen.

show abstract

Recommendations for Increasing Replicability in Psychology

Cited by 752 publications

References 48 publications

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

Bayesian tests to quantify the result of a replication attempt.

Wege aus der Vertrauenskrise – Individuelle Schritte hin zu verlässlicher und offener Forschung

Contact Info

Product

Resources

About