Improving the Replicability of Psychological Science Through Pedagogy

Abstract

Replications are important to science, but who will do them? One proposal is that students can conduct replications as part of their training. As a proof-of-concept for this idea, here we report a series of 11 pre-registered replications of findings from the 2015 volume of Psychological Science, all conducted as part of a graduate-level course.Congruent with larger, more systematic efforts, replications typically yielded smaller effects than originals: The modal outcome was partial support for the original claim.This work documents the challenges facing motivated students as they attempt to replicate previously published results on a first attempt. We describe the workflow and pedagogical methods that were used in the class and discuss implications both for the adoption of this pedagogical model and for replication research more broadly.Keywords: Replication; Reproducibility; Pedagogy; Experimental Methods REPLICATION THROUGH PEDAGOGY 3 Improving the Replicability of Psychological Science Through PedagogyReplicability is a core value for empirical research and there is increasing concern throughout psychology that more independent replication is necessary (Open Science Collaboration, 2015; Wagenmakers, Wetzels, Borsboom, Maas, & Kievit, 2012). Yet under the current incentive structure for science, replication is not typically valued for publication in top journals (Makel, Plucker, & Hegarty, 2012) or in metrics of research productivity (Koole & Lakens, 2012). One potential solution to this problem is to make replication an explicit part of pedagogy: that is, to teach students about experimental methods by asking them to run replication studies (Frank & Saxe, 2012; Grahe et al., 2012). Despite enthusiasm for this idea (Everett & Earp, 2015; M. King et al., 2016;LeBel, 2015;Standing, 2016), there is limited data beyond anecdotal reports and individual projects (Lakens, 2013; e.g., Phillips et al., 2015) to support its efficacy in producing wide-scale pedagogical adoption.In the current article, we describe the pedagogical and methodological approach to replication research taken in our graduate-level experimental methods course and address the practical barriers faced by instructors planning to incorporate replications into their courses. In our course, students conducted replications of published articles from the 2015 volume of the journal Psychological Science with rigorous instructor review at each major stage. The results of these replications are a microcosm of larger replication efforts, providing insight into both the difficulties of pedagogical replications and their promise as a method for improving the robustness of psychological research.We assess the challenges facing a student in choosing an article of interest and -in a single attempt, within constraints of budget, expertise, and effort -reproducing the findings. We consider a number of criteria for evaluating replication success, including statistical significance, effect size, a Bayesian measure of evidence (Etz & REPLICATION THROUG...

Similar event-related potentials to structural violations in music and language

“…Finally, we note that there has been a growing interest in conducting serious replication studies in undergraduate and graduate research methods classes (Frank & Saxe, 2012;Grahe, Brandt, IJzerman, & Cohoon, 2014;Hawkins et al, 2018;Wagge, Baciu, Banas, Nadler, & Schwarz, 2019;Wagge, Brandt, et al, 2019). The hypothesized benefits are numerous: students act as real scientists with tangible outcomes, motivating careful and engaged work on the part of the students and benefiting the scientific community with the generation of new evidence; students learn about the mechanics and process of conducting scientific research with well-defined research questions and procedures, providing a stronger foundation for generating novel research in the future; reading papers with the goal of replication teaches students to critically evaluate the methods and rationales in order to be able to replicate the work (Frank & Saxe, 2012).…”

Learning Open Science by doing Open Science. A reflection of a qualitative research project-based seminar

“…The term replication crisis refers to the difficulties involved in reproducing results from scientific studies (Randall & Welser, 2018). The authors (Frank & Saxe, 2012;Grahe et al, 2012;Hawkins et al, 2018) argue replications are timeconsuming and expensive, and "normal" researchers are often unwilling to do this task. One response to these challenges is for students to conduct replication studies with open data (Frank & Saxe, 2012;Hawkins et al, 2018;Toelch & Ostwald, 2018).…”

How stable are preferences among emerging electricity generation technologies

“…MTurk is a large online platform on which registered 'requesters' (task creators) can post tasks and recruit registered 'workers' (paid task completers) to participate [26]. Previous studies comparing behavioral experiments using MTurk and other recruitment methods have found few systematic differences in how people respond to decision-making tasks [27,28]. For example, a recent research project replicated 15 experiments to examine the generalizability of MTurk results, for manipulations such as framing and priming.…”