Participant carelessness and fraud: Consequences for clinical research and potential solutions.

Abstract

Clinical psychological research studies often require individuals with specific characteristics. The Internet can be used to recruit broadly, enabling the recruitment of rare groups such as people with specific psychological disorders. However, Internet-based research relies on participant self-report to determine eligibility, and thus, data quality depends on participant honesty. For those rare groups, even low levels of participant dishonesty can lead to a substantial proportion of fraudulent survey responses, and all studies will include careless respondents who do not pay attention to questions, do not understand them, or provide intentionally wrong responses. Poor-quality responses should be thought of as categorically different from high-quality responses. Including these responses will lead to the overestimation of the prevalence of rare groups and incorrect estimates of scale reliability, means, and correlations between constructs. We demonstrate that for these reasons, including poor-quality responses-which are usually positively skewed-will lead to several data-quality problems including spurious associations between measures. We provide recommendations about how to ensure that fraudulent participants are detected and excluded from self-report research studies.

The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

“…Although MTurk operates an internal rating system of their workers, researchers are strongly advised to embed various attention and validity checks in their data collection tools to verify and ensure the quality of the collected data (Chandler, Shapiro, & Sisso, 2019; Dunn, Heggestad, Shanock, & Theilgard, 2018). Our own review of the methodological studies on MTurk revealed various types of data quality measures, including infrequency items (which have only one correct or highly probable answer), time measurements (with a minimum reading speed threshold), person-total correlations (capturing the subject’s internal consistency relative to the expected patterns derived from all other participants), long string analyses (which flags participants with a long string of identical answers), and exclusion of non-U.S. Internet protocol (IP) addresses and ones that are suspected to be indicative of automated activity.…”

The quality of research on mental health related to the COVID-19 pandemic: A note of caution after a systematic review

“…The low compliance with open science recommendations found in our review, although it is in line with previous findings in psychology ( Vanpaemel et al., 2015 ) and psychiatry ( Sherry et al., 2020 ), showing that this practice is poorly followed by researchers, confirms that research related to the COVID-19 does not follow recommended measures to increase replicability of findings. Additionally, some warnings and recommendations have been made about the quality of internet-collected responses ( Chandler et al., 2020 ) and the practice of pre-analysis plans ( Olken, 2015 ) that apparently are not followed in the studies analysed.…”

Are you for real? Maximizing participant eligibility on Amazon's Mechanical Turk

“…This includes screenshots of human intelligence tasks (HITs), study eligibility criteria, embedded attention checks and survey links. Among all five intervention studies that our team has conducted on MTurk to date, we were able to find evidence of this information sharing on TurkOpticon, Reddit, MTurkCrowd, TurkView and mturkforum [7][8][9][10]. Therefore, it is recommended that researchers monitor these common websites regularly during recruitment.…”