Eating disorder prevalence among Amazon MTurk workers assessed using a rigorous online, self-report anthropometric assessment

Concerns and recommendations for using Amazon MTurk for eating disorder research

“…Finally, early MTurk reviews were promising, suggesting the platform was not only convenient and inexpensive, but also resulted in high-quality data that appeared as reliable as traditional data collection methods (Behrend et al, 2011;Buhrmester et al, 2011). As researchers both outside of and within the ED field continue to publish using MTurk samples, and voice confidence in their data's validity (Kambanis, Bottera, & De Young, 2021), we were excited by MTurk's potential to help us obtain high-quality data from a hard-to-access population.…”

“…The research team convened and concluded, given the vast majority of our data were of questionable quality, we could not ethically disseminate or interpret analyses from this sample. However, given MTurk is the most popular online crowdsourcing platform for researchers (Dennis et al, 2020), and continues to be widely used in the ED field (Carr et al, 2019;Kambanis et al, 2021;Lydecker et al, 2018), we felt a responsibility to share our experience and offer caution and recommendations to other ED researchers.…”

“…Although it is well-known that sampling bias is a risk of MTurk because workers are often not representative of the general population (Keith et al, 2017), one notable trend is that MTurk workers show elevated rates of psychopathology generally (Arditte, Çek, Shaw, & Timpano, 2016;Jensen-Doss, Patel, Casline, Mora Ringle, & Timpano, 2021;Keith et al, 2017;Shapiro et al, 2013), and ED symptomatology specifically (Kambanis et al, 2021). For instance, Arditte et al (2016) found that MTurk workers reported mood and anxiety symptoms at levels comparable to those with clinical disorders.…”

“…Furthermore, a recent integrative review found that MTurk participants were less emotionally stable, and had lower well-being, greater negative affect, and more negative attitudes than other samples (Keith et al, 2017). Rather than view these findings as concerning for potential participant misrepresentation or lack of data integrity, however, researchers often conclude that evidence of elevated psychopathology could make MTurk a particularly valuable tool for collecting data from clinical populations (Kambanis et al, 2021;Shapiro et al, 2013). Our findings suggest those conclusions may be more complicated and raise questions about the integrity of published MTurk data.…”

“…In a study conducted during COVID-19, Kambanis et al (2021) explored rates of EDs and ED behaviors in a sample of MTurk workers. The authors employed a rigorous process to mitigate invalid data, including validity/attention checks, >75% HIT rates, duplicate response monitoring, and evaluation of completion time (although participants completing the survey in <60% of median completion time were retained).…”