Revisiting Mental Simulation in Language Comprehension: Six Replication Attempts

Abstract

The notion of language comprehension as mental simulation has become popular in cognitive science. We revisit some of the original empirical evidence for this. Specifically, we attempted to replicate the findings from earlier studies that examined the mental simulation of object orientation, shape, and color, respectively, in sentence-picture verification. For each of these sets of findings, we conducted two web-based replication attempts using Amazon's Mechanical Turk. Our results are mixed. Participants responded faster to pictures that matched the orientation or shape implied by the sentence, replicating the original findings. The effect was larger and stronger for shape than orientation. Participants also responded faster to pictures that matched the color implied by the sentence, whereas the original studies obtained mismatch advantages. We argue that these results support mental simulation theory, show the importance of replication studies, and show the viability of web-based data collection.

Open Science: What, Why, and How

“…Though falsifying data was rated as neither prevalent nor defensible, a few practices were prevalent (e.g., failing to report all dependent measures, reporting only studies that confirmed the hypothesis, data peeking), and many were viewed as defensible (John, Loewenstein, & Prelec, 2012 Kerr (1998) reports that in an unpublished study, over half of his respondents said that editors had asked them to alter a hypothesis post hoc. OPEN SCIENCE 8 Pashler, 2013;Johnson, Cheung, & Donnellan, 2014;LeBel & Campbell, 2013;Lynott et al, 2014;McDonald, Donnellan, Lang, & Nikolajuk, 2014;Pashler, Coburn, & Harris, 2012;Shanks et al, 2013;Zwaan & Pecher, 2012 widely shared e-mail that with regards to that research area: "I see a train wreck looming" (Kahneman, 2012).…”

“…At the same time that psychologists were acknowledging the prevalence of QRPs, researchers across the field had become more vocal about being unable to replicate others' findings, including some that were well-cited and ground-breaking (Donnellan, Lucas, & Cesario, 2015;Doyen, Klein, Pichon, & Cleeremans, 2012;Harris, Coburn, Rohrer, & Pashler, 2013;Johnson, Cheung, & Donnellan, 2014;LeBel & Campbell, 2013;Lynott et al, 2014;McDonald, Donnellan, Lang, & Nikolajuk, 2014;Pashler, Coburn, & Harris, 2012;Shanks et al, 2013;Zwaan & Pecher, 2012). Although there have always been failures to replicate, now many of them were occurring in multiple laboratories and, because of new media, there was increasing recognition that the failures were not isolated to single labs or lines of research.…”

Participant Nonnaiveté and the reproducibility of cognitive psychology

“…We put a sample of these findings to a particularly stringent test by running them under circumstances that are increasingly representative of current practices of data collection but also are documented as challenging for reproducibility. In particular, we conducted the first preregistered replication of a large set of cognitive psychological effects in the most popular online participant pool (Crump, McDonnell, &Gureckis, 2013 andPecher, 2012 for non-preregistered replications on MTurk). Most importantly, we examined whether reproducibility depends on participant nonnaïveté by conducting the same experiments twice on the same participants a few days apart.…”

Why the A/AN prediction effect may be hard to replicate: A rebuttal to DeLong, Urbach & Kutas (2017)

“…We think this opposition is visible in psycholinguistics, too. Conceptual and direct replication research is unfortunately very sparse (but see Nieuwland et al, 2017;Rommers, Meyer, & Huettig, 2013;Zwaan & Pecher, 2012; see also Jäger, Engelmann, & Vasishth, 2017), and even novel but incremental contributions are often considered insufficient for publication.…”