Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

Abstract

Background Amazon Mechanical Turk (MTurk) is an online labor market in which people ("requesters") requiring the completion of small tasks ("Human Intelligence Tasks" [HITs]) are matched with people willing to do them ("workers"). MTurk has become a popular data collection tool among social science researchers: In 2015, the 300 most influential social science journals (with impact factors greater than 2.5, according to Thomson-Reuters InCites) published more than 500 articles that relied on MTurk data in full or in part (Chandler & Shapiro, 2016). Reflecting the popularity of MTurk, considerable effort has been invested in evaluating data collected from it, with particular emphasis on documenting the demographic and psychological characteristics of its population, the quality of respondent data, and the methodological limitations of the platform. As a result, MTurk workers have become one of the most thoroughly studied convenience samples currently available to researchers (for a review, see Chandler & Shapiro, 2016), and researchers have learned a great deal about the ways in which MTurk respondents are and are not similar to the general population. There are reasons to suspect, however, that there are also important variations between different samples drawn from MTurk, and these variations have received far less attention. This article addresses this question, using data from a study of approximately 10,000 MTurk workers to examine whether sample composition varies as a function of the time that it is collected. We begin by reviewing what extant research reveals about the demographic composition of the MTurk worker pool. Then, we describe the methods and measures that we use in our study, after which we present the results of our analyses, 712774S GOXXX10.

Common Concerns with MTurk as a Participant Pool: Evidence and Solutions

“…However, it is difficult to assess the impact of these posts without knowing how likely any particular MTurker is to see a post that is relevant to a study they have not yet completed or how discussion frequency in forums compares to the amount of crosstalk among physical lab participants. In one research study about 10% of participants reported finding a study on a site outside of MTurk (Casey et al, 2017), though most forums have strong norms against discussing study content (Chandler, Mueller, & Paolacci 2014). Note that, like most concerns in this chapter, worries about nonnaivete are not unique to only MTurk, as crosstalk has also been observed within college subject pools (Edlund, Sagarin, Skowronski, Johnson, & Kutter, 2009).…”

“…However, averages can be deceiving because experience follows a power law, with a small proportion of extremely active users and a long tail of less active users (Chandler et al, 2014;Rand et al, 2014). Active MTurkers tend to find and complete studies quickly (Casey et al, 2017), leading them to be both very experienced and overrepresented in studies of MTurker knowledge.…”

“…Researchers should also monitor MTurk forums in order to verify that MTurkers are not sharing compromising information about the study (e.g., turkernation.com; mturkforum.com; reddit.com/r/hitsworthturkingfor; Hauser & Schwarz, 2016;Casey et al, 2017). Google seems to have good indices of these forums, so researchers can monitor by googling their requester name and the name of their HIT in order to find crosstalk.…”

“…Typically less than 5% of participants submit responses from IP addresses that are shared with other participants. Some of these appear to be proxy servers used by ISPs, and of the remainder, about half of the responses seem to be generated by different people in the same household completing the survey (Chandler et al, 2014;Casey et al, 2017). Recently, Dennis and colleagues (2018) found evidence that some workers use virtual private servers to participate in multiple studies.…”

Effects of Participant Displeasure on the Social-Psychological Study of Power on Amazon’s Mechanical Turk

“…One limitation of this research is that it is not representative of the entire MTurk population. This is due to workers self-selecting into each study, more experienced workers enrolling more quickly (Arechar, Kraft-Todd, & Rand, 2017;Casey, Chandler, Levine, Proctor, & Strolovitch, 2017), and because I required workers to meet or exceed a minimum "reputation" (i.e., percent of work deemed acceptable by previous requesters), which, to be meaningful, required workers to have completed a minimum number of HITs (Peer, Vosgerau, & Acquisti, 2014). A possible disadvantage of not examining relatively inexperienced workers is that displeasure in recall-based methods may partly be a function of repeated exposure.…”