Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

Chandler, Jesse; Mueller, Pam; Paolacci, Gabriele

doi:10.3758/s13428-013-0365-7

Cited by 890 publications

(784 citation statements)

References 57 publications

Supporting

Mentioning

757

Contrasting

Unclassified

Order By: Relevance

“…Science is cumulative however, and instead of seeking to repeat these demonstrations here, we proceeded to use this scenario to test the previously theorized motivational underpinnings of this effect. The fact that this scenario produced a weaker effect than previously observed could be a result of any of several factors, including the observation that psychological effects become smaller as the participant population becomes more experienced in psychological tasks (34,35), or that participants were given more time to reflect on their responses than in previous experiments. In any case, if the present scenario constitutes a relatively weaker manipulation of the variable of interest, it provides a conservative platform for testing our main hypotheses.…”

Section: Preliminary Experimentscontrasting

confidence: 43%

Perceived intent motivates people to magnify observed harms

Ames

Fiske

2015

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Existing moral psychology research commonly explains certain phenomena in terms of a motivation to blame. However, this motivation is not measured directly, but rather is inferred from other measures, such as participants' judgments of an agent's blameworthiness. The present paper introduces new methods for assessing this theoretically important motivation, using tools drawn from animal-model research. We test these methods in the context of recent "harm-magnification" research, which shows that people often overestimate the damage caused by intentional (versus unintentional) harms. A preliminary experiment exemplifies this work and also rules out an alternative explanation for earlier harm-magnification results. Exp. 1 asks whether intended harm motivates blame or merely demonstrates the actor's intrinsic blameworthiness. Consistent with a motivational interpretation, participants freely chose blaming, condemning, and punishing over other appealing tasks in an intentional-harm condition, compared with an unintentional-harm condition. Exp. 2 also measures motivation but with converging indicators of persistence (effort, rate, and duration) in blaming. In addition to their methodological contribution, these studies also illuminate people's motivational responses to intentional harms. Perceived intent emerges as catalyzing a motivated social cognitive process related to social prediction and control.intent | motivation | harm | blame | morality

show abstract

Section: Preliminary Experimentscontrasting

confidence: 43%

Perceived intent motivates people to magnify observed harms

Ames

Fiske

2015

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Although we were primarily concerned about suspicion in experiments in which the purpose of the perspective-taking task was relatively transparent and performance was easily alterable, we decided to impose a similar suspicion exclusion rule across experiments. Suspicion was generally low across experiments; we suspect that it was higher among MTurk users because of their greater experience with experiments (particularly autobiographical recall emotion inductions), relative to college students (Chandler, Mueller, & Paolacci, 2014). Analyses including these participants' data are reported in Table Appendix A. Additionally, in Experiment 1, we excluded data from participants who provided unscorable location descriptions on the spatial perspective-taking task (e.g., "at the top").…”

Section: Resultsmentioning

confidence: 99%

Anxious and egocentric: How specific emotions influence perspective taking.

Todd

Forstmann

Burgmer

et al. 2015

Journal of Experimental Psychology: General

135

108

View full text Add to dashboard Cite

People frequently feel anxious. Although prior research has extensively studied how feeling anxious shapes intrapsychic aspects of cognition, much less is known about how anxiety affects interpersonal aspects of cognition. Here, we examine the influence of incidental experiences of anxiety on perceptual and conceptual forms of perspective taking. Compared with participants experiencing other negative, high-arousal emotions (i.e., anger or disgust) or neutral feelings, anxious participants displayed greater egocentrism in their mental-state reasoning: They were more likely to describe an object using their own spatial perspective, had more difficulty resisting egocentric interference when identifying an object from others' spatial perspectives, and relied more heavily on privileged knowledge when inferring others' beliefs. Using both experimental-causal-chain and measurement-of-mediation approaches, we found that these effects were explained, in part, by uncertainty appraisal tendencies. Further supporting the role of uncertainty, a positive emotion associated with uncertainty (i.e., surprise) produced increases in egocentrism that were similar to anxiety. Collectively, the results suggest that incidentally experiencing emotions associated with uncertainty increase reliance on one's own egocentric perspective when reasoning about the mental states of others.

show abstract

“…This is particularly true of MTurk for which there is a growing concern that respondents have evolved to be less and less like respondents in other surveys (even survey panels). 13 Rand et al (2014) report that in MTurk data collected between February 2011 and February 2013, the median MTurk respondent reported participation in 300 academic studies, 20 of which were in the last week; moreover, they note that, over the time period they studied, "the MTurk subject pool [had] transformed from naïve to highly experienced… [and this] makes it likely that subjects will be familiar more generally with experimental paradigms…" (4-5; also see Chandler, Mueller, and Paolacci 2014).…”

Section: Discussionmentioning

confidence: 99%

The Generalizability of Survey Experiments

Mullinix¹,

Leeper²,

Druckman³

et al. 2015

J Exp Polit Sci

1,003

603

View full text Add to dashboard Cite

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

show abstract

Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

Cited by 890 publications

References 57 publications

Perceived intent motivates people to magnify observed harms

Perceived intent motivates people to magnify observed harms

Anxious and egocentric: How specific emotions influence perspective taking.

The Generalizability of Survey Experiments

Contact Info

Product

Resources

About