Taking the time? Explaining effortful participation among low-cost online survey participants

Abstract

Recent research has shown that Amazon MTurk workers exhibit substantially more effort and attention than respondents in student samples when participating in survey experiments. In this paper, I examine when and why low-cost online survey participants provide effortful responses to survey experiments in political science. I compare novice and veteran MTurk workers to participants in Qualtrics’s qBus, a comparable online omnibus program. The results show that MTurk platform participation is associated with substantially greater effort across a variety of indicators of effort relative to demographically-matched peers. This effect endures even when compensating for the amount of survey experience accumulated by respondents, suggesting that MTurk workers may be especially motivated due to an understudied self-selection mechanism. Together, the findings suggest that novice and veteran MTurk workers alike are preferable to comparable convenience sample participants when performing complex tasks.

The shape of and solutions to the MTurk quality crisis

“…1 Though met with skepticism by some, MTurk respondents tend to yield high-quality data when respondents are screened on reputation (Peer et al ., 2014). In fact, MTurk samples generally provide higher quality data than student samples, community samples, and even some high-quality national samples (Hauser and Schwarz, 2015; Mullinix et al ., 2015; Thomas and Clifford, 2017; Anson, 2018). For these reasons, the use of MTurk for survey research has grown dramatically across a variety of disciplines, including psychology (Paolacci and Chandler, 2014; Zhou and Fishbach, 2016), economics (Horton et al ., 2011), public administration (Stritch et al ., 2017), and sociology (Shank, 2016).…”

Catholics and capital punishment: Do Pope Francis’s teachings matter in policy preferences?

“…This national-level opt-in platform allows "workers" to identify and participate in a variety of tasks for a small incentive-in our case, $2.30 for completing an online survey. Online opt-in surveys are useful for producing more honest and accurate self-reports (i.e., less social desirability bias, satisficing, speeding, interviewer effects), which is important given the topic of study (Anson, 2018, Chang & Krosnick, 2009, Weinberg, Freese, & McElhattan, 2014. Following listwise deletion using the same approach as the YouGov samples, the sample was reduced from 1,000 initial respondents to an analytic sample of 983 respondents.…”

Faith in Trump, Moral Foundations, and Social Distancing Defiance During the Coronavirus Pandemic

“…Amazon's MTurk platform allows eligible "workers" to select and perform various tasks for a small financial incentive-in this case, $2.30 for completing an online survey. Compared to other data collection modes, online opt-in surveys result in more honest and accurate self-reports (less social desirability bias, less satisficing, less speeding, and no interviewer effects) (Anson, 2018;Chang & Krosnick, 2009;Weinberg, Freese, & McElhattan, 2014). Eligible respondents for our study were MTurk workers 18 years old or older who lived in the United States and who, for high-quality respondents, had completed over 500 previous HITS (human intelligence tasks) and had a 95% or higher approval rating (Peer, Vosgerau, & Acquisti, 2014).…”