Intertemporal Differences Among MTurk Workers

Abstract

The online labor market Amazon Mechanical Turk (MTurk) is an increasinglypopular platform for generating samples of respondents for social scienceresearch. A growing body of research has examined the demographiccomposition of MTurk workers, typically comparing samples of MTurk workersto samples of respondents drawn from other populations. While thesecomparisons have revealed important information about the ways in whichMTurk workers are and are not representative of the general population,variations among samples drawn from MTurk have received less attention.This paper focuses on whether MTurk sample composition varies as a functionof time. Using an original dataset of nearly 10,000 MTurk workers, weexamine whether demographic characteristics vary by (1) time of day, (2)day of week, and (3) serial position (i.e., earlier or later in datacollection). We find that day of week differences are minimal, but thattime of day and serial position are associated with small but importantvariations in demographic composition, including characteristics known toimpact political attitudes and psychological processes. This demonstratesthat MTurk samples cannot be presumed identical across different studies,and we suggest several forms of variation to which researchers using MTurkought to be attentive.

Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk

“…With respect to the non-incentivized measures, a comparison between our results and those of Casey et al (2016) reveals substantial convergence: both papers find more experienced participants earlier in the day and earlier in the data collection process, that participants who scored lower on the Big-5 personality dimension of conscientiousness were more likely to complete HITs later in the day, and that participants tended to score higher in the Big-5 personality dimension of agreeableness earlier in the data collection process.…”

“…There is evidence in support of such heterogeneity; for example, people who work in traditional white collar jobs may be unavailable to complete studies during regular business hours. As a result, studies run during those hours may be more likely to recruit “professional” participants who use MTurk as a primary source of income – and thus may have more prior experience (Casey et al 2016), make fewer errors (Chandler et al 2015), and complete studies more quickly (Deetlefs et al 2015). Additionally, participants recruited when a study is first posted may differ from those recruited later, as in college samples where there is evidence that students differ depending on whether they sign up to complete studies at the beginning versus the end of the semester (Aviv et al 2002).…”

“…Additionally, participants recruited when a study is first posted may differ from those recruited later, as in college samples where there is evidence that students differ depending on whether they sign up to complete studies at the beginning versus the end of the semester (Aviv et al 2002). Indeed, in an unincentivized survey study, Casey et al (2016) explore the demographic and personality differences of participants who took part in surveys at different times on MTurk. Notably, they find that experienced participants were more likely to complete tasks earlier in the day, and that participants tend to be older, less neurotic and more conscientious earlier in the data collection.…”

Moving outside the lab: The viability of conducting sensorimotor learning studies online

“…These "professional" participants typically make less errors and complete studies more efficiently (Casey & Chandler, 2016;Chandler, Paolacci, Peer, Mueller, & Ratliff, 2015;Hauser, Paolacci, & Chandler, 2019. Thus, we limited the time of data collection to regular business hours by launching the experiment in the morning around 9 am California time.…”

Reliability of web-based affective auditory stimulus presentation

“…Given the recent COVID-19 pandemic (Wang et al, 2020), online experimental studies may now even be necessary to circumvent restrictions in carrying out in-lab testing. Web-based testing is attractive not only for its convenience, but it also offers a wealth of advantages ranging from its cheap and rapid data collection system to reach large sample sizes necessary for well-powered research (Gillan & Daw, 2016) to the availability of more diverse or under-represented populations (Berinsky et al, 2012; Casey et al, 2018; Goodman et al, 2013; Levay et al, 2016; Majima et al, 2017; Shapiro et al, 2013). Moreover, there is strong evidence that web collected data is qualitatively on par with data collected from traditional participant pools (Berinsky et al, 2012; Klein et al, 2014; Paolacci et al, 2010), with high internal reliability and test-retest reliability (Shapiro et al, 2013).…”