A conceptual replication examining the risk of overtly listing eligibility criteria on Amazon’s Mechanical Turk

Abstract

Recent scholarship indicates that explicitly listing eligibility requirements on Amazon’s Mechanical Turk can lead to eligibility falsification. Offering a conceptual replication of prior studies, we assessed the prevalence of eligibility falsification and its impact on data integrity. A screener survey collected the summer before the 2016 presidential election assessed political affiliation. Participants were then randomly assigned to be exposed to a second survey link for which they were eligible or ineligible. There was a significant interaction such that the differences between self‐reported Republicans and Democrats on outcome measures (e.g., attitudes toward Hillary Clinton), were smaller among participants that were falsifying eligibility (i.e., imposters) than those that were not (i.e., genuine participants). Moreover, for most outcomes, imposters put forth responses that were significantly different from the responses put forth by those in the political party with which imposters were pretending to be affiliated. Imposters’ responses were also significantly different from participants in the political party with which imposters initially claimed to genuinely belong. For example, those who self‐reported themselves as Democrats on the screener survey but responded to a survey for “only Republicans” (i.e., imposter Republicans), reported more favorable attitudes toward Donald Trump than genuine Democrats, but indicated less favorable attitudes toward Donald Trump than genuine Republicans. These results highlight the potential harms of explicitly listing eligibility requirements and emphasize the importance of minimizing imposter participation.

Data Collection via Online Platforms: Challenges and Recommendations for Future Research

“…In addition, researchers have found across multiple studies that between 2.2 and 28 percent of participants on MTurk misrepresent their qualifications, even after utilizing best practice measures to ensure data quality (MacInnis et al, 2020). Fraudulent and dishonest behavior is even higher when researchers have over‐restrictive participation criteria or aim to recruit specialist samples such as full‐time employees, entrepreneurs or professionals from a certain industry (Chandler & Paolacci, 2017; Siegel & Navarro, 2019; Siegel, Navarro, & Thomson, 2015; Wessling, Huber, & Netzer, 2017).…”

Are you for real? Maximizing participant eligibility on Amazon's Mechanical Turk

“…This includes screenshots of human intelligence tasks (HITs), study eligibility criteria, embedded attention checks and survey links. Among all five intervention studies that our team has conducted on MTurk to date, we were able to find evidence of this information sharing on TurkOpticon, Reddit, MTurkCrowd, TurkView and mturkforum [7][8][9][10]. Therefore, it is recommended that researchers monitor these common websites regularly during recruitment.…”

“…We offer four practical solutions to address this concern. The first solution is to avoid providing explicit eligibility criteria when recruiting through MTurk [8]. This approach requires a delicate balance between providing participants with sufficient information to make an informed decision without making the eligibility criteria obvious.…”

“…Because MTurk includes a relatively high base-rate of people who endorse clinical symptoms [4,5], it provides a good vehicle for collecting these samples. Recently, studies have reported that an alarmingly high number of people (upwards of 71.9%) misrepresent important characteristics to meet eligibility criteria [6][7][8][9]. Misrepresentation, which appears to be increasing, is a grave concern.…”

Communicating Reform: Testing an Apology for Police History as a Supplement to a Policy Communication

“…MTurk staff claims over 200,000 respondents in the United States (Robinson et al, 2019). Of those, estimates for African American respondents vary between 5.8% and 8.1% (Jeong et al, 2019;Siegel & Navarro, 2019;Walter et al, 2019). MTurk is more reliable than other online panels when respondents are experienced, there are checks to ensure that respondents are paying attention, and respondents are not allowed to re-take surveys (Christenson & Glick, 2013;Hauser & Schwarz, 2016;Kees et al, 2017a;Paas et al, 2018;Paolacci & Chandler, 2014).…”