Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Ebersole, Charles R.; Atherton, Olivia E.; Belanger, Aimee Lynn; Skulborstad, Hayley M; Allen, Jill N.; Banks, Jonathan; Baranski, Erica; Bernstein, Michael J.; Bonfiglio, Diane B. V.; Boucher, Leanne; Brown, Elizabeth R.; Budiman, Nancy I.; Cairo, Athena H.; Capaldi, Colin A.; Chartier, Christopher R.; Cicero, David C.; Coleman, Jennifer A.; Conway, Morgan Anna; Davis, William E.; Devos, Thierry; Dopko, Raelyne L.; Grahe, Jon; German, Komi; Hicks, Joshua A.; Hermann, Anthony D.; Humphrey, Brandon T.; Johnson, David J.; Joy-Gaba, Jennifer A.; Juzeler, Hannah; Klein, Richard Anthony; Lucas, Richard E.; Lustgraaf, Christopher J. N.; Menon, Madhavi; Metzger, Mitchell M.; Moloney, Jaclyn M.; Morse, Patrick; Nelson, Anthony J.; Prišlin, Radmila; Razza, Timothy; Ré, Daniel; Rule, Nicholas O.; Sacco, Donald F.; Sauerberger, Kyle; Shultz, Megan Rose; Smith, Jessi Lyn; Sobocko, Karin; Steiner, Troy G.; Sternglanz, Rolf; Tskhay, Konstantin O.; Vaughn, Leigh Ann; Allen, Zack van; Walker, Ryan J.; Wilson, John Paul; Wirth, James H.; Wortman, Jessica; Zelenski, John M.; Nosek, Brian A.

doi:10.31234/osf.io/q4emc

Cited by 141 publications

(240 citation statements)

References 15 publications

(24 reference statements)

Supporting

Mentioning

226

Contrasting

Unclassified

Order By: Relevance

“…Several studies, however, have tried to conduct a similar experiment, one that examined the influence of weight on aspects related to seriousness (Jostmann et al, 2009 (Ebersole et al, 2016). In response to these failures to replicate, Jostmann, Schubert, and Lakens (2016) commented: "We have had to conclude that there is actually no reliable evidence for the effect" (p. 93).…”

Section: Discussionmentioning

confidence: 99%

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)

Beek¹,

Matzke²,

Pinto³

et al. 2017

Preprint

View full text Add to dashboard Cite

This preregistered replication attempt focuses on the finding from Ackerman, Nocera, & Bargh (2010; ANB) that holding a heavy object triggers concepts related to importance. ANB reported that participants who were holding a heavy clipboard rated a job candidate as better overall and more seriously interested in the job than participants holding a light clipboard. We failed to replicate ANB's results; instead, Bayes factor hypothesis tests indicated evidence for the absence of a difference between the heavy and the light condition in the overall evaluation and perceived seriousness of the candidate, and in participants' perceived task importance. Our findings highlight the importance of conducting preregistered replication research and illustrate the theoretical and practical advantages of Bayesian inference in psychological research.

show abstract

Section: Discussionmentioning

confidence: 99%

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)

Beek¹,

Matzke²,

Pinto³

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…Ebersole et al (2016) found little heterogeneity across labs, and a small effect for the time of the semester (which was the main moderator investigated in the project), which seemed to suggest that context matters little in determining the outcome of Stroop performance.…”

Section: Random Forests and The Many Labs Initiative -The Stroop Effectmentioning

confidence: 99%

“…A number of recent articles were published with replications, many of which failed to replicate. One example of this is known as "ManyLabs3" (Ebersole et al, 2016). In many of these reports, authors conclude that context matters little, which in ManyLabs3 was concluded due to little heterogeneity across labs (but see IJzerman et al, 2016).…”

Section: What Predicts Stroop Performance? a Conditional Random Foresmentioning

confidence: 99%

See 1 more Smart Citation

What Predicts Stroop Performance? A Conditional Random Forest Approach

et al. 2016

View full text Add to dashboard Cite

An experimental science relies on solid and replicable results. The last few years have seen a rich discussion on the reliability and validity of psychological science and whether our experimental findings can falsify our existing theoretical models. Yet, concerns have also arisen that this movement may impede new theoretical developments. In this article, we reanalyze the data from a crowdsourced replication project that concluded that lab site did not matter as predictor for Stroop performance, and, therefore, that there were no "hidden moderators" (i.e., context was likely to matter little in predicting the outcome of the Stroop task). The authors challenge this conclusion via a new analytical method-supervised machine learning -that "allows the data to speak." The authors apply this approach to the results from a Stroop task to illustrate the utility of machine learning and to propose moderators for future (confirmatory) testing. The authors discuss differences with some conclusions of the original article, which variables need to be controlled for in future inhibitory control tasks, and why psychologists can use machine learning to find surprising, yet solid, results in their own data. What Predicts Stroop Performance? A Conditional Random Forest ApproachAn experimental science relies on solid and replicable results. The last few years have seen a rich discussion on the reliability and validity of psychological science and whether our experimental findings can falsify our existing theoretical models (Coyne, 2016). Arguably, one of the most important events calling into question the solidity of psychological science was the publication on precognition by Daryl Bem (2011) followed by a paper on data contingent analyses by Simmons, Nelson, and Simonsohn (2011). Following these papers, the times, they have been a-changing: Extensive discussions have been held on power, replicability, pre-registration, and context sensitivity (Bakker, Hartgertink, Wicherts, & Van der Maas, in press;Brandt, IJzerman et al., 2014, Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016IJzerman et al., 2016). Though Bem's (2011) paper was perhaps the most prominent, problems seem to be widespread, and these problems are often blamed on the fact that novelty has been championed over solid and replicable findings (Coyne, 2016;Nosek, Spies, & Motyl, 2012). Yet, this begs the question: How can we retain creative and innovative exploration in service of finding out "truths" about human functioning? Currently, the most popular way to do so is to conduct an experiment, explore the data, write a paper, replicate this study, and, if it fails, update one's theoretical assumptions (and perhaps argue on social media for a while). This approach is incredibly sensitive to post hoc interpretation and is inefficient. Indeed, there are near infinite reasons why effects replicate or not and failures to replicate often do little to inform theoretical predictions (but, see Brandt, IJzerman, et al., 2014). A number of recent articles were publishe...

show abstract

“…Specifically, IJzerman and Koole (2011) explained that this aspect of their theory was not true for warmth related effects (see also Lee and Schwarz, 2012; for skepticism of this view), as they discuss findings showing that relationship warmth has an effect on the perception of ambient temperature (for original studies and replications, see Zhong and Leonardelli, 2008;IJzerman and Semin, 2010;Szymkow et al, 2013;IJzerman et al, 2015a; for a notable non-replication, see Ebersole et al, 2015). As an alternative to Lakoff and Johnson's model, IJzerman and Koole (2011) identified Barsalou's (1999, 2008 perceptual symbol systems (PSSs) as a rival candidate for attachment-related processes regarding underlying psychological processes in which concepts are built in perceptual systems through simulations relying on perceptual elements.…”

Section: Is Attachment Radically Embodied?mentioning

confidence: 99%

Toward a Radically Embodied Neuroscience of Attachment and Relationships?

Beckes¹,

IJzerman²

2017

Preprint

View full text Add to dashboard Cite

Attachment theory (Bowlby, 1969(Bowlby, /1982 posits the existence of internal working models as a foundational feature of human bonds. Radical embodied approaches instead suggest that cognition requires no computation or representation, favoring a cognition situated in a body in an environmental context with affordances for action (Chemero, 2009;Barrett, 2011;Wilson and Golonka, 2013;Casasanto and Lupyan, 2015). We explore whether embodied approaches to social soothing, interpersonal warmth, separation distress, and support seeking could replace representational constructs such as internal working models with a view of relationship cognition anchored in the resources afforded to the individual by their brain, body, and environment in interaction. We review the neurobiological bases for social attachments and relationships and attempt to delineate how these systems overlap or don't with more basic physiological systems in ways that support or contradict a radical embodied explanation. We suggest that many effects might be the result of the fact that relationship cognition depends on and emerges out of the action of neural systems that regulate several clearly physically grounded systems. For example, the neuropeptide oxytocin appears to be central to attachment and pair-bond behavior (Carter and Keverne, 2002) and is implicated in social thermoregulation more broadly, being necessary for maintaining a warm body temperature (for a review, see IJzerman et al., 2015b). Finally, we discuss the most challenging issues around taking a radically embodied perspective on social relationships. We find the most crucial challenge in individual differences in support seeking and responses to social contact, which have long been thought to be a function of representational structures in the mind (e.g., Baldwin, 1995). Together we entertain the thought to explain such individual differences without mediating representations or computations, but in the end propose a hybrid model of radical embodiment and internal representations.

show abstract

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Abstract: Many Labs 3 is a crowdsourced project that systematically evaluated time-of-semester effects across many participant pools. See the Wiki for a table of contents of files and to download the manuscript.

Cited by 141 publications

References 15 publications

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)

What Predicts Stroop Performance? A Conditional Random Forest Approach

Toward a Radically Embodied Neuroscience of Attachment and Relationships?

Contact Info

Product

Resources

About

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Abstract: Many Labs 3 is a crowdsourced project that systematically evaluated time-of-semester effects across many participant pools. See the Wiki for a table of contents of files and to download the manuscript.

Cited by 141 publications

References 15 publications

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, &amp; Bargh (2010)

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, &amp; Bargh (2010)

What Predicts Stroop Performance? A Conditional Random Forest Approach

Toward a Radically Embodied Neuroscience of Attachment and Relationships?

Contact Info

Product

Resources

About

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)

Incidental Haptic Sensations May Not Influence Social Judgments: A Purely Confirmatory Replication Attempt of Study 1 by Ackerman, Nocera, & Bargh (2010)