Practice makes perfect: Commentary on Burnette et al. (2021)

Abstract

Underrepresented identities have been overlooked in the development of measures assessing eating disorders; therefore, limited normative data exist for these identities.To address this, Burnette et al. sought to provide Eating Disorder Examination-Questionnaire and Eating Attitudes Test-26 norms for transgender adults using Amazon's MTurk. However, they were unable to achieve this goal due to what they perceived as high rates of invalid responses. Instead, they provided recommendations for conducting MTurk research. However, little or no evidence supports the validity of several recommendations, partly because their study was not designed to derive or validate recommendations. By their own admission, their strategies failed to address what they identified as the central problem. We express concern about Burnette et al.'s recommendations because (a) the recommendations are built on assumptions about the problem that may not be true; and (b) the recommendations are not provided within the context of limitations of self-report/online data collection writ large. We detail these concerns and propose that strategies for mitigating inattentive/invalid responding be subjected to validation prior to being recommended to prevent the implementation of procedures that result in the exclusion of the target population, individuals who we historically, and perhaps still, unjustly exclude from research.

The Relationship of Self-identified Weight Status With Perceived Mental and Physical Health

“…We also used Amazon's Mechanical Turk website to recruit participants for the current study. There is debate over the ethics of using this site as a source of participants, including questions about the validity of data, [32][33][34][35][36] although survey data from participants recruited through Mechanical Turk do not appear notably worse than those obtained through other internet surveys. [37][38][39] To protect our data from potential invalid responses, we included 3 formal validity checks throughout the survey and examined response patterns during the data cleaning process.…”

Eating Disorder Examination–Questionnaire and Clinical Impairment Assessment norms for intersectional identities using an MTurk sample

“…Amazon Mechanical Turk (MTurk), a crowdsourcing platform used for clinical research (Keith et al, 2017), provides access to research participation for large samples of underrepresented groups (e.g., Paolacci et al, 2010) that may be difficult to recruit through standard methods. Although there are challenges to ensuring the validity of data collected using MTurk (Burnette et al, 2021), many are not unique to this platform (De Young & Kambanis, 2021) and generally effective strategies (e.g., cross‐referencing data for inconsistent responding, utilizing open‐ended questions, assessing response time; Behrend et al, 2011) are available to mitigate them. Moreover, researchers are using MTurk to recruit individuals for studies about EDs (e.g., Arditte et al, 2016; Carr et al, 2019; Lydecker & Grilo, 2017; Pollert et al, 2016).…”