Teaching Replication

Frank, Michael C.; Saxe, Rebecca

doi:10.1177/1745691612460686

Cited by 171 publications

(94 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The American Association for Applied Linguistics (2017) recently amended its guidelines to recommend that "high quality replication studies, which are critical in many domains of scientific inquiry within applied linguistics, be valued on par with non-replication-oriented studies." Engaging students with conducting replication studies has also been discussed (see Frank & Saxe, 2012;Porte, 2012), and we are aware of several graduate programs where replication is an integral part of training and assessment.…”

Section: Wider Cultural Changes In Academiamentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

Despite its critical role for the development of the field, little is known about replication in second language (L2) research. To better understand replication practice, we first provide a narrative review of challenges related to replication, drawing on recent developments in psychology. This discussion frames and motivates a systematic review, building on syntheses of replication in psychology, education, and L2 research. We coded 67 self-labeled L2 replication studies found across 26 journals for 136 characteristics. We estimated a mean rate of 1 published replication study for every 400 articles, with a mean of 6.64 years between initial and replication studies and a mean of 117 citations of the initial study before a replication was published. Replication studies had an annual mean of 7.3 citations, much higher than averages in linguistics and education. Overlap in authorship between initial and replication studies and the availability of the initial materials both increased the likelihood of a replication supporting the initial findings. Our sample contained no direct (exact) replication attempts, and changes made to initial studies were numerous and wide ranging, which likely obscured, if not undermined, the interpretability of replication studies. To improve the amount and quality of L2 replication research, we propose 16 recommendations relating to rationale, nomenclature, design, infrastructure, and incentivization for collaboration and publication.

show abstract

Section: Wider Cultural Changes In Academiamentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

show abstract

“…For example, replications are said to be able to protect science from fraud and questionable research practices (Crocker & Cooper, 2011) and clarify ambiguous results (Simmons et al, 2011). Replication is called "the gold standard for reliability" and "even if a small number of [independent replications] find the same result, then that result can be relied on" (Frank & Saxe, 2012). Finally, replications are supposed to uncover false positives that are the result of publication bias (Diekmann, 2011;Murayama, Pekrun, & Fiedler, 2013).…”

Section: Why Do We Want More Observations and More Studies?mentioning

confidence: 99%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

“…Such training provides a vast opportunity to collect replication data which, with reasonable oversight to ensure data quality, could be compiled into large replication databases (Grahe et al, 2012). Teachers who have already begun incorporating replications into their methods classes report that students are excited to contribute to cutting-edge research beyond the classroom, produce quality data, and benefit from focusing on the details of study design and procedures (Frank & Saxe, 2012;Grahe et al, 2012).…”

Section: Practicing Open Science: For Teachersmentioning

confidence: 99%

Open Science: What, Why, and How

Spellman¹,

Gilbert²,

Corker³

2017

Preprint

View full text Add to dashboard Cite

Open science is a collection of actions designed to make scientific processes more transparent and results more accessible. Its goal is to build a more replicable and robust science; it does so using new technologies, altering incentives, and changing attitudes. The current movement toward open science was spurred, in part, by a recent series of unfortunate events within psychology and other sciences. These events include the large number of studies that have failed to replicate and the prevalence of common research and publication procedures that could explain why. Many journals and funding agencies now encourage, require, or reward some open science practices, including pre-registration, providing full materials, posting data, distinguishing between exploratory and confirmatory analyses, and running replication studies. Individuals can practice and promote open science in their many roles as researchers, authors, reviewers, editors, teachers, and members of hiring, tenure, promotion, and awards committees. A plethora of resources are available to help scientists, and science, achieve these goals. Keywords: data sharing, file drawer problem, open access, open science, preregistration, questionable research practices, replication crisis, reproducibility, scientific integrityThanks to Brent Donnellan (big thanks!), Daniël Lakens, Calvin Lai, Courtney Soderberg, and Simine Vazire OPEN SCIENCE 3 Open Science When we (the authors) look back a couple of years, to the earliest outline of this chapter, the open science movement within psychology seemed to be in its infancy. Plenty of people were pointing to problems in psychology research, collecting archival data to support the claims, and suggesting how science could be improved. Now it seems that the open science movement has reached adolescence. Things are happening-and they are happening quickly. New professional organizations are being formed to uncover and facilitate ways to improve science, often through new technology, and some old organizations are adopting new procedures to remedy problems created by past practices, often involving revising journal policies. Researchers are changing the way they teach, practice, and convey science. And scientific information (and opinions) are traveling fast. In blogs, tweets, Facebook groups, op eds, science journalism, circulation of preprints, post-print comments, video talks, and so on, more people are engaged in communicating science, and hoping to improve science, than ever before. Thus, any new technology, new procedure, new website, or new controversy we describe is likely to be superseded (or solved) even by the time this chapter is published. But the core values of open science should remain. The "Open Science" MovementScience is about evidence: observing, measuring, collecting, and analyzing evidence. And it is about evidence that can be shared across observers and, typically, although not necessarily exactly (Merton, 1973;Popper, 1959), replicated later. Science is about testing hypotheses, using inductiv...

show abstract

Teaching Replication

Cited by 171 publications

References 22 publications

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Open Science: What, Why, and How

Contact Info

Product

Resources

About