Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality

Barends, Ard J.; Vries, Reinout E. de

doi:10.1016/j.paid.2019.02.015

Cited by 99 publications

(80 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Participants (N = 332, M age = 37.07, SD age = 11.36, 36% female, 79% United States) 1 were recruited from Amazon's MTurk and provided monetary compensation. Prior research has supported the validity of findings obtained from MTurk participants, and we applied exclusion guidelines from these sources to ensure sufficient data quality ( Barends & de Vries, 2019 ; Buchheit, Dalton, Pollard, & Stinson, 2019 ). Only participants that had completed more than 50 MTurk tasks with greater than 95% lifetime approval were included.…”

Section: Methodsmentioning

confidence: 99%

Who wants to reopen the economy during the COVID-19 pandemic? The daring and uncaring

Howard

2021

Personality and Individual Differences

View full text Add to dashboard Cite

Economic shutdowns, which refer to disallowing employees to work on site, are among the most contentious approaches to reduce the spread of COVID-19. While economic shutdowns save lives, their large economic costs have caused some people to develop strong attitudes and even break government-issued mandates, which incurs health risks and often the need to extend the economic shutdowns. In the current article, we argue that the interaction of two personality characteristics, risk-taking tendencies and prosocial tendencies, is a strong determinant of attitudes toward economic shutdowns, and we assess the impact of this interaction on three different attitudes toward economic shutdowns that differ by their focal target: employees, customers, and organizations. The results demonstrate that this interaction significantly predicted economic shutdown attitudes toward customers and organizations but not employees. We suggest that these results can be understood via the lens of behavioral decision-making theories as well as a recent framework on antisocial risk takers, both of which provide several subsequent directions for future research. We conclude with recommendations for the development of effective messages to curb defiant behaviors toward economic shutdowns, such as focusing on those most likely to perform these problematic behaviors – the daring and uncaring.

show abstract

Section: Methodsmentioning

confidence: 99%

Who wants to reopen the economy during the COVID-19 pandemic? The daring and uncaring

Howard

2021

Personality and Individual Differences

View full text Add to dashboard Cite

show abstract

“… Note. Sources used to derive recommendations: 1 Antin and Shaw (2012); 2 Arechar, Gächter, and Molleman (2018); 3 Barends and de Vries (2019); 4 Bederson and Quinn (2011); 5 Behrend, Sharek, Meade, and Wiebe (2011); 6 Bergvall-Kåreborn and Howcroft (2014); 7 Brawley and Pury (2016); 8 Buchanan and Scofield (2018); 9 Buhrmester, Talaifar, and Gosling (2018); 10 Chandler, Mueller, and Paolacci (2014); 11 Chandler, Paolacci, Peer, Mueller, and Ratliff (2015); 12 Cheung, Burns, Sinclair, and Sliter (2017); 13 Clifford and Jerit (2014); 14 Deng, Joshi, and Galliers (2016); 15 Feitosa, Joseph, and Newman (2015); 16 Fieseler, Bucher, and Hoffmann (2017); 17 Gleibs (2017); 18 Goodman, Cryder, and Cheema (2013); 19 Hydock (2018); 20 Kan and Drummey (2018); 21 Litman, Robinson, and Rosenzweig (2015); 22 Mummolo and Peterson (2019); 23 Necka, Cacioppo, Norman, and Cacioppo (2016); 24 Wessling, Huber, and Netzer (2017); 25 Zhou and Fishbach (2016). HIT = human intelligence task. …”

Section: Literature Reviewmentioning

confidence: 99%

“…Many MTurker responses are unusable due to high attrition rates and MTurker inattention. Therefore, in addition to the sample size determined through a power analysis, it is useful to collect data from at least an additional 15% to 30% of MTurkers (Sprouse, 2011) to compensate for participant attrition and failure to pass attention and compliance checks (Barends & de Vries, 2019; Zhou & Fishbach, 2016).…”

Section: Recommendationsmentioning

confidence: 99%

“…The use of web-based research using Amazon’s Mechanical Turk (MTurk) has increased tenfold over just the last decade (Walter, Seibert, Goering, & O’Boyle, 2019), making it by far the most frequently used online data collection method (Porter, Outlaw, Gale, & Cho, 2019). Despite its popularity, there are concerns that call into question the validity of research conclusions based on MTurk data (e.g., Barends & de Vries, 2019; Hydock, 2018; Zack, Kennedy, & Long, 2019). These concerns are severe enough that some journals have intermittently refused to accept manuscripts that utilized MTurk, and some journal editors and reviewers have summarily recommended rejecting manuscripts that used MTurk regardless of a study’s other positive features (Landers & Behrend, 2015; Walter et al, 2019).…”

mentioning

confidence: 99%

See 1 more Smart Citation

MTurk Research: Review and Recommendations

Aguinis

Villamor

Ramani

2020

Journal of Management

637

420

View full text Add to dashboard Cite

The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.

show abstract

“…We excluded participants who failed our pre-registered attention checks, but the data quality still appeared poor compared to the undergraduate sample included in Study 1 and when compared to samples reported in prior research using similar measures (Siegelman et al, 2018). There is some suggestion in the literature that attention checks can be limited in effectiveness (Chandler et al, 2014;Hauser & Scharz, 2016) and that other statistical approaches may be needed to identify engaged participants (e.g., Barends & de Vries, 2019;Dunn et al, 2018). Indeed, it is highly plausible that the seeming passiveness of the task (i.e., clicking on images as the appeared) rendered our experiment susceptible to participant distraction.…”

Section: Emotion and Statistical Learningmentioning

confidence: 99%

Emotional congruency facilitates statistical learning

Plate¹,

Schapiro²,

Waller³

2020

Preprint

View full text Add to dashboard Cite

Detecting regularities and extracting patterns is a vital skill that helps us to organize complex information in our environments. Statistical learning, a process through which we detect regularities by attending to the relationships between cues in our environment, contributes to knowledge acquisition across myriad domains. However, less is known about how emotional cues—specifically facial configurations of emotion—influence statistical learning. These cues pose challenges for statistical learning because they can vary both inter- and intra-individually and across contexts. Thus, the degree of congruence between the emotional and statistical information may diverge in important ways. In the current study, we tested two pre-registered aims to advance knowledge about emotional congruency and statistical learning: (1) We examined statistical learning of emotional information when compared to non-emotional information, and (2) We assessed how congruency influences regularity extraction. Using data from two independent samples (total N=671; 49% female, 50% male, 1% “other” or did not report gender; <1% American Indian or Alaskan Native; 14% Asian, 14% Black or African American, 5% Hispanic or Latinx; 4% Multiracial; 63% White), we demonstrated statistical learning across emotional cues similar to non-emotional cues (i.e., shapes). We also established that congruent cues facilitated statistical learning but found little evidence that incongruent information interfered with statistical learning. The results shed light on the application of a well-studied cognitive skill to the social domain and fill a knowledge gap about how cue congruency impacts statistical learning, including how we use emotional cues to make predictions in our social world.

show abstract

Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality

Cited by 99 publications

References 25 publications

Who wants to reopen the economy during the COVID-19 pandemic? The daring and uncaring

Who wants to reopen the economy during the COVID-19 pandemic? The daring and uncaring

MTurk Research: Review and Recommendations

Emotional congruency facilitates statistical learning

Contact Info

Product

Resources

About