Are samples drawn from Mechanical Turk valid for research on political ideology?

Abstract

Amazon's Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the national public, we know little about whether liberals and conservatives recruited from MTurk share the same psychological dispositions as their counterparts in the mass public. In the absence of such evidence, some have argued that the selection process involved in joining MTurk invalidates the subject pool for studying questions central to political science. In this paper, we evaluate this claim by comparing a large MTurk sample to two benchmark national samples -one conducted online and one conducted face-to-face. We examine the personality and value-based motivations of political ideology across the three samples. All three samples produce substantively identical results with only minor variation in effect sizes. In short, liberals and conservatives in our MTurk sample closely mirror the psychological divisions of liberals and conservatives in the mass public, though MTurk liberals hold more characteristically liberal values and attitudes than liberals from representative samples. Overall, our results suggest that MTurk is a valid recruitment tool for psychological research on political ideology.

Social and economic ideologies differentially predict prejudice across the political spectrum, but social issues are most divisive.

“…Recent research indicates that MTurk is a valid platform for conducting research on political ideology (Clifford, Jewell, & Waggoner, 2015), and that MTurk results are comparable to those drawn from nationally representative samples (Brandt & van Tongeren, in press). Participants first provided perceived warmth ratings on 0 -100 scales for 18 (Sample 2) and 39 (Sample 3) targets, with 41 unique targets across Samples 2 and 3.…”

Past-focused environmental comparisons promote proenvironmental outcomes for conservatives

“…MTurk samples tend to produce results that are as valid and reliable as laboratory-based samples (23) and have been shown to be as representative of the United States population as other sampling methods used in political science research (24)(25)(26). To avoid the most critical problem with Mturk samples-nonnaiveté (27)-participants were barred from taking part in more than one of our studies.…”

Online erotica usage as a mediator between internet addiction and engagement in risky online sexual behaviors

“…More specifically, data gathered multiple scales from MTurk samples have alpha levels in the good to excellent range, and test-retest reliabilities are also acceptable (Buhrmester et al, 2011). Additionally, there is much empirical evidence that MTurk participants perform similarly on various measures (including priming and behavioral tasks, and measures of emotional reactions, personality, and political ideation) to those recruited via other standard recruitment methods, like college samples or through social media (e.g., Casler, Bickel, & Hackett, 2013;Clifford, Jewell, & Waggoner, 2016;Eriksson & Simpson, 2010;Feitosa, Joseph, & Newman, 2015;Horton, Rand, & Zeckhauser, 2011). Moreover, MTurk samples have been used specifically to conduct research in the field of sexuality (e.g., Séguin, Milhausen, & Kukkonen, 2015;Webster & Crysel, 2012).…”

“…It would be helpful, in future, to explore whether internet addiction leads to compulsive use of pornography or sex sites (using validated measures for compulsivity), and whether these compulsive behaviors are also predictive of risky online sexual behaviors. Second, we included only MTurk participants, who although shown to be similar to other samples in measures of behavior and personality (e.g., Casler et al, 2013;Clifford et al, 2016;Eriksson & Simpson, 2010;Feitosa et al, 2015;Horton et al, 2011) may be different from college samples or those recruited via other methods. Additionally, we included only U.S. residents, who may be different from those residing in other nations, especially in terms of the prevalence of online sexual behaviors.…”