Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones &amp; Leister (2018): Charitable giving across donor types

Gandullia, Luca; Lezzi, Emanuela; Parciasepe, Paolo

doi:10.1016/j.joep.2020.102268

Cited by 18 publications

(11 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, for certain characteristics like overall cognitive ability, or beliefs about privacy and security, MTurk samples have been found to be representative of national samples (Merz et al, 2020 ; Redmiles et al, 2019 ). MTurk has also been used successfully to replicate experimental psychology research for which results of the original, offline experiment are considered valid (Ganduillia et al, 2020 ). Nonetheless, crowdsourced sampling is different because of the explicitly transactional nature of the process (e.g., “workers” completing a “task” rather than “participants in a research study”).…”

Section: Quality Of Crowdsourced Samples From Mturkmentioning

confidence: 99%

Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

et al. 2021

View full text Add to dashboard Cite

Crowdsourced psychological and other biobehavioral research using platforms like Amazon’s Mechanical Turk (MTurk) is increasingly common – but has proliferated more rapidly than studies to establish data quality best practices. Thus, this study investigated whether outcome scores for three common screening tools would be significantly different among MTurk workers who were subject to different sets of quality control checks. We conducted a single-stage, randomized controlled trial with equal allocation to each of four study arms: Arm 1 (Control Arm), Arm 2 (Bot/VPN Check), Arm 3 (Truthfulness/Attention Check), and Arm 4 (Stringent Arm – All Checks). Data collection was completed in Qualtrics, to which participants were referred from MTurk. Subjects (n = 1100) were recruited on November 20–21, 2020. Eligible workers were required to claim U.S. residency, have a successful task completion rate > 95%, have completed a minimum of 100 tasks, and have completed a maximum of 10,000 tasks. Participants completed the US-Alcohol Use Disorders Identification Test (USAUDIT), the Patient Health Questionnaire (PHQ-9), and a screener for Generalized Anxiety Disorder (GAD-7). We found that differing quality control approaches significantly, meaningfully, and directionally affected outcome scores on each of the screening tools. Most notably, workers in Arm 1 (Control) reported higher scores than those in Arms 3 and 4 for all tools, and a higher score than workers in Arm 2 for the PHQ-9. These data suggest that the use, or lack thereof, of quality control questions in crowdsourced research may substantively affect findings, as might the types of quality control items.

show abstract

Section: Quality Of Crowdsourced Samples From Mturkmentioning

confidence: 99%

Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Importantly, because MT has been increasingly used for purposes of research data collection, several studies have tried to investigate the accuracy and reliability of such datasets. In particular, several studies have shown that the results of famous economic (including the "Linda" problem) (Paolacci et al, 2010) and social (Brañas-Garza et al, 2020;Gandullia et al, 2020) experiments can be replicated using MT subjects' pools. Furthermore, it has been showed that participants in MT show similar levels of attention and buy into interactive experiments (Thomas & Clifford, 2017), as well as the same level of generalizability of experimental findings (Coppock, 2019) as compared to more traditional subject pools.…”

Section: The Experiments On Mechanical Turkmentioning

confidence: 99%

The Effect of Reputation Concerns on the Conjunction Fallacy: A Field Experiment on Facebook

Devetag

Calluso

2023

Preprint

View full text Add to dashboard Cite

“…When quality control approaches are utilized, data from MTurk can broadly mirror the distribution of population-level data for risk assessments like the US Alcohol Use Disorders Identification Test [ 34 , 53 ]. In addition, scholars have successfully replicated ‘in-person’ experimental research using MTurk [ 54 ]. Thus, it is reasonable to infer some level of generalizability, especially given the nationally representative draw of race, gender, and age cross-sections.…”

Section: Limitationsmentioning

confidence: 99%

Beliefs and misperceptions about naloxone and overdose among U.S. laypersons: a cross-sectional study

2022

View full text Add to dashboard Cite

Background Overdose education and naloxone distribution (OEND) to laypersons are key approaches to reduce the incidence of opioid-involved overdoses. While some research has examined attitudes toward OEND, especially among pharmacists and first responders, our understanding of what laypersons believe about overdose and naloxone is surprisingly limited. Further, some scholars have expressed concerns about the prevalence of non-evidence-based beliefs about overdose and naloxone. We designed this study to analyze the prevalence, nature, and context of beliefs about naloxone and overdose among U.S. laypersons. Methods We conducted a cross-sectional study (n = 702) using Prolific.co (representative of the U.S. population by age, gender, and race). Primary outcomes were the believability of six statements about overdose/naloxone on a seven-point Likert-type scale. Five statements were unsupported, and one was supported, by current scientific evidence. We used latent profile analysis to classify participants into belief groups, then used regression to study correlates of profile classification. Results Believability of the statements (7: extremely believable) ranged from m = 5.57 (SD = 1.38) for a scientifically supported idea (trained bystanders can reverse overdose with naloxone), to m = 3.33 (SD = 1.83) for a statement claiming opioid users can get high on naloxone. Participants were classified into three latent belief profiles: Profile 1 (most aligned with current evidence; n = 246), Profile 2 (moderately aligned; n = 351), and Profile 3 (least aligned, n = 105). Compared to Profile 1, several covariates were associated with categorization into Profiles 2 and 3, including lower trust in science (RRR = 0.36, 95%CI = 0.24–0.54; RRR = 0.21, 95%CI = 0.12–0.36, respectively), conservative political orientation (RRR = 1.41, 95%CI = 1.23–1.63; 3:RRR = 1.62, 95%CI = 1.35–1.95, respectively), and never being trained about naloxone (Profile 3: RRR = 3.37, 95%CI = 1.16–9.77). Conclusions Preliminary evidence suggests some U.S. laypersons simultaneously believe that bystander overdose prevention with naloxone can prevent overdose and one or more scientifically unsupported claims about naloxone/overdose. Categorization into clusters displaying such belief patterns was associated with low trust in science, conservative political orientation, and not having been trained about naloxone. Preregistration This cross-sectional study was preregistered prior to any data collection using the Open Science Framework: https://osf.io/c6ufv

show abstract

Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types

Cited by 18 publications

References 35 publications

Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

The Effect of Reputation Concerns on the Conjunction Fallacy: A Field Experiment on Facebook

Beliefs and misperceptions about naloxone and overdose among U.S. laypersons: a cross-sectional study

Contact Info

Product

Resources

About