A Primer for Conducting Survey Research using MTurk

Abstract

This paper presents best practices for conducting survey research using Amazon Mechanical Turk (MTurk). Readers will learn the benefits, limitations, and trade-offs of using MTurk as compared to other recruitment services, including SurveyMonkey and Qualtrics. A synthesis of survey design guidelines along with a sample survey are presented to help researchers collect the best quality data. Techniques, including SPSS and R syntax, are provided that demonstrate how users can clean resulting data and identify valid responses for which workers could be paid.

Crowdsourcing for Online Samples in Counseling Research

“…In addition to efficiency in time, crowdsourcing platforms are inexpensive options. Although exact costs depend on the particular platform, the eligibility criteria of the sample, and the length of the survey, the average cost per participant is about $0.70 for MTurk, $5 for SurveyMonkey, and $7 for Qualtrics (Chambers et al, 2016). As long as compensation is realistic, it does not affect data quality-particularly with MTurk, in which the researcher sets the rate (Buhrmester et al, 2011).…”

“…As noted earlier, much research relies on easy-to-access samples, such as undergraduate and graduate students. Although scholars agree there is varying understanding around the diversity of the demographics, crowdsourcing platforms are more likely Crowdsourcing for Online Samples in Counseling Research to attract younger individuals (Chambers et al, 2016;Huff & Tingley, 2015). Platform participants also tend to report higher education levels, lower income, less religious affiliation, and more introversion compared with national norms (Burnham et al, 2018;Paolacci et al, 2010).…”

“…However, despite these oversampled demographics, the samples derived from crowdsourcing appear to be more similar to the U.S. population than are traditional university subject pools (Buhremester et al, 2011;Paolacci et al, 2010). Some platforms also let researchers acquire international samples (Chambers et al, 2016). Researchers can develop diverse samples by setting screening procedures and/or identifying who should be recruited for a study.…”

“…Participants may lie on demographic data or on sampling criteria to ensure inclusion in the study (Skitka & Sargis, 2005). Tech-savvy subjects can participate multiple times in a single study (Chambers et al, 2016). Crowdsourcing platforms have attempted to control for repeat participation, such as Qualtrics's "prevent ballot box stuffing" option, which is based on internet browser cookies that are easy to work around.…”

“…Also, researchers must ensure they are knowledgeable of institutional and legal policies surrounding participant compensation. For example, rejecting worker payment may violate institutional review board guidelines or laws (Chambers et al, 2016).…”