Collecting Online Survey Data: A Comparison of Data Quality among a Commercial Panel & MTurk

Abstract

Scholars seeking online data collection have sought the assistance of professional panels, which are commercial entities that recruit prospective research participants. Although not intended for recruiting participants, Amazon's Mechanical Turk (MTurk) service is a crowdsourcing platform that is being utilized for a lower cost. This study compares data collected from MTurk (N = 760) and a professional panel (N = 1,110) to assess aspects of data quality and respondent features. Results show that MTurk may produce better data quality based on completion rate and success in passing manipulation checks. Participants also differ in terms of educational attainment, age, and preferences of certain devices used for participation. Overall, results provide insight that researchers collecting online data should take into consideration before choosing a data collection platform. Researchers collecting online data increasingly utilize professional panels (Antoun et al. 2016). Opt-in panels allow researchers to cost-effectively recruit respondents compared to offline recruitment (Antoun et al. 2016). Amazon's Mechanical Turk (MTurk) crowdsourcing platform provides an alternate avenue for participant recruitment in exchange for payment (Hitlin 2016). This study compares data collected from MTurk and a professional panel to assess data quality, respondent features, and devices used for responding. Results showcase practical implications for acquiring valid data when recruiting online. Literature Review Data Collection Online participant recruitment has been adopted for their convenience, efficiency, and low cost (Paolacci, Chandler, and Ipeirotis 2010). Two popular means of online recruitment include professional panel companies and crowdsourcing websites, such as MTurk.

Comparing Online Surveys for Cybersecurity: SONA and MTurk

“…However, through SONA or MTurk (at home, over the internet), participants may feel less compelled to answer in a socially desirable manner, resulting in more accurate responses. It has also been shown that MTurk participants are slightly more demographically diverse [16,[18][19][20], which is something that we saw in our survey responses in terms of a wider age range for participants. However, the results from SONA and in-person surveys were similar [16].…”

Association between Dog Owner Demographics and Decision to Seek Veterinary Care

“…Participants were recruited for an online anonymous survey utilizing the Amazon Mechanical Turk (mTurk) crowdsourcing marketplace via Amazon.com. Previous research has indicated that mTurk samples provide socioeconomically and culturally diverse participants [25] and data quality are comparable to, or superior to, other platforms used to obtain survey participants [25][26][27]. For these reasons, mTurk was chosen in an effort to recruit a sample of dog owners that resembled the parameters of the United States demographic population.…”

Accelerating the Adoption of Disruptive Technologies: The Impact of COVID-19 on Intentions to Use Autonomous Vehicles

“…This bias towards Left-leaning partisanship is not, however, attributed to the higher response rates from states that tend to vote Democrat in recent elections, an observation confirmed by simulating elections using the sample distribution and election data (43). Instead, this bias is likely the result of selfselection in non-random online surveys (44)(45)(46). Finally, the average and median income for the sample are $72,100 and $62,500 annually, compared to $88,600 and $62,800 in the 2019 5-year American Community Survey (42).…”