Crowdsourcing Consumer Research

Abstract

Data collection in consumer research has progressively moved away from traditional samples (e.g., university undergraduates) and toward Internet samples. In the last complete volume of the Journal of Consumer Research (June 2015–April 2016), 43% of behavioral studies were conducted on the crowdsourcing website Amazon Mechanical Turk (MTurk). The option to crowdsource empirical investigations has great efficiency benefits for both individual researchers and the field, but it also poses new challenges and questions for how research should be designed, conducted, analyzed, and evaluated. We assess the evidence on the reliability of crowdsourced populations and the conditions under which crowdsourcing is a valid strategy for data collection. Based on this evidence, we propose specific guidelines for researchers to conduct high-quality research via crowdsourcing. We hope this tutorial will strengthen the community’s scrutiny on data collection practices and move the field toward better and more valid crowdsourcing of consumer research.

“Keep on Turkin’”?

“…Use of crowdsourcing destinations such as MTurk to collect data provides a variety of advantages when compared to traditional samples (Goodman and Paolacci 2017). These include (1) reduced costs, (2) quick response, (3) greater participant diversity, (4) superior data quality, and (5) greater research flexibility.…”

“…Moreover, this diversity can range across customer segments, industries, even cultures. As a result, with careful study design researchers can potentially increase both internal and external validity (Goodman and Paolacci 2017).…”

“…Although many researchers use MTurk to source experimental subjects, some scholars have suggested use of MTurk workers across a broader range of research designs. For example, Goodman and Paolacci (2017) suggest that use of MTurk can facilitate longitudinal studies, cross-cultural research, interactions between participants, and use of alternative measures.…”

“…Launched by Amazon in 2005, MTurk is Ban online crowdsourcing labor market^ (Levay et al 2016). In their review of work published over four years in the Journal of Consumer Research (volumes 39-42), Goodman and Paolacci (2017) observed that 27% of all surveys and experiments in the journal were conducted using MTurk. Even more strikingly, this proportion of MTurk-based empirical work grew from 9% in the first of the four years they reviewed to 43% in the final year.…”

“…1 This explosion in the use of MTurk has fostered interest in understanding the nature of MTurk Bworkers^and assessing the extent to which the data they provide can be considered reliable and valid (e.g., Berinsky et al 2012;Goodman et al 2013). The results of this assessment are mixed, as discussed in detail in an excellent review by Goodman and Paolacci (2017). Perhaps most importantly, the issue of economic motivation for MTurk workers to misrepresent themselves has been found to be an important determinant of data quality (Wessling et al 2017).…”