The Cyborg Method: A Method to Identify Fraudulent Responses from Crowdsourced Data

Abstract

Crowdsourcing platforms have grown in popularity in the last decade. This approach has become an essential data collection method, especially when access to participant pools are limited by location and resources. Concerns about the validity and quality of crowdsourced data persist, however. A recent documented increase in the amount of invalid responses within crowdsourced data has highlighted the need for quality control measures. The present study evaluated the utility of a Cyborg Method that involved two components – an automated evaluation of participant meta-data and a human review of open-ended responses. Two samples were recruited – the first applied the Cyborg Method after data collection to gauge the extent that invalid responses would be collected without such measures. The second applied the method during data collection to determine the method’s ability to proactively screen invalid responses and reduce study costs such that invalid responses were not given compensation. Results suggested that Cyborg Method identified a substantial portion of invalid responses. The use of both automated and human evaluation components was necessary to screen out these responses. Furthermore, the application of the Cyborg Method during data collection substantially reduced the per participant cost of data collection. These results suggest that the Cyborg Method is a promising means by which to maintain the quality of crowdsourced data.

COVID‐19 related distress and sleep health among trauma‐exposed South Asians: Does generational status matter?

“…Of those initial responses, 3096 total participants were excluded for any of the following reasons to ensure data integrity (see Figure 1 ): (1) did not meet inclusionary criteria, (2) failed to pass all three validity checks, (3) had duplicate responses, (4) had inconsistent generational status, immigration status, and/or country of birth responses, (5) had >30% item‐level missing data on any of the key measures, or (6) had invalid internet protocol (IP) addresses. Invalid IP addresses were flagged by an automated IP evaluation service which determined if the IP address of the participant was associated with the use of a Virtual Private Network or a robot (BOT) network ( Price et al., in review ). We used IPHub (iphub.info) to determine if a participant had a US non‐residential IP address (hosting provider, proxy, etc.)…”

“…that suggested the IP address was either a blocked or bad IP address. IPHub was selected because it has been shown to perform well in prior studies in which IP evaluations were used (Dennis et al., 2020; Price et al., in review). Other studies using MTurk samples have taken similar approaches to ensure data integrity, and show that such approaches enhance data quality and validity (Forkus et al., 2021; Hauser & Schwarz, 2016; Paolacci et al., 2010; Price et al., in review; Tobar‐Santamaria et al., 2021).…”

Survivor vs. Victim: Self-Labeling of Trauma Victims and Its Implicit Impact on Resilience and Recovery

“…This sample is also a limitation considering the possible inclusion of robotic answers (Webb, 2022). Though we screened the data to identify people who merely clicked through to finish the survey, this screening was visual and though an IP (Internet Protocol) screener as recommended by Price (2022), so more robotic answers could have been missed. The variability in test environment, participant motivation and possible inclusion of robotic answers could have resulted in variability of scores outside the normal fluctuation (Chandler et al, 2020, Shapiro et al, 2013.…”