Using Online Labor Market Participants for Nonprofessional Investor Research: A Comparison of MTurk and Qualtrics Samples

Abstract

Recently, researchers have begun using online labor markets to recruit participants for experimental studies examining the judgments and decisions of nonprofessional investors. This study investigates the quality and generalizability of data collected from these sources by replicating an experimental task from Elliott, Hodge, Kennedy, and Pronk (2007) using nonprofessional investor participants from two popular online labor markets—Amazon's Mechanical Turk (MTurk) and Qualtrics Online Sample (Qualtrics). Compared to Qualtrics participants, we find that MTurk participants pay greater attention to the experimental materials and better acquire and recall information. Further, the MTurk sample more closely replicates EHKP's investment club member results on measures of information integration than does the Qualtrics sample. These results provide some evidence that many interesting research questions can be satisfactorily answered using nonprofessional investor participants from MTurk. We believe further investigation is needed before Qualtrics can be endorsed as a high-quality source of nonprofessional investor participants.

How to Mitigate Security-Related Stress: The Role of Psychological Capital

“…We collected 150 data sets by distributing an online questionnaire over crowdsourcing marketplace Amazon MTurk, which is no longer an exception in scientific research [51]. Data collected via online labor markets are externally and internally valid [7].…”

Investment Experience, Financial Literacy, and Investment‐Related Judgments

“…MTurk, which has recently become a popular source of participants for social scientists (Brandon et al ; Ipeirotis ), presents a potential solution to this obstacle. Data from MTurk participants has recently been applied in published accounting research (e.g., Rennekamp ), with concurrent research replicating extant accounting studies with MTurk participants to assess the validity of using this participant pool, either as an initial step (Koonce et al ) or as the main contribution of the research (Farrell et al ; Owens and Hawkins ). My research contributes to this developing trend in accounting research by explicitly benchmarking the sample of investors recruited from MTurk against nationwide samples selected to be representative of U.S. Census distributions (FINRA Foundation , ).…”

“…Third, this research extends the accounting literature on the effect of individual cognitive characteristics on judgment and decision making (e.g., Bonner 2008;Libby and Luft 1993) by examining the impact of investment experience and financial literacy on individual investors' judgments. Prior research (including concurrent research replications with MTurk participantse.g., Koonce et al 2015;Farrell et al 2017;Owens and Hawkins 2018) has generally not assessed the conditions under which results predicated on effortful processing of financial information would replicate with a range of backgrounds and skill sets. My findings suggest that assessing investment experience and financial literacy can help to identify individuals who are able and willing to study financial reporting information with reasonable diligence.…”

Why People Don’t Use Facebook Anymore? An Investigation Into the Relationship Between the Big Five Personality Traits and the Motivation to Leave Facebook

“…Originally, a total of 503 participants were recruited from the online survey system Amazon Mechanical Turk (mturk). This system is known for its high quality and generalizability of data (Owens and Hawkins, 2018), and researchers confirm that data collected from mturk is demographically diverse (Buhrmester et al, 2011), and provides reliable results that are consistent with decision-making research (Goodman et al, 2013). The IRB approved this study before the data collection, and all participants electronically signed consent forms.…”