The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

Stewart, Neil; Ungemach, Christoph; Harris, Adam J. L.; Bartels, Daniel M.; Newell, Ben R.; Paolacci, Gabriele; Chandler, Jesse

doi:10.1017/s1930297500005611

Cited by 214 publications

(33 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The present studies also relied on participants recruited through Mturk. Recent work has noted that Mturk participants may have prior experience with canonical judgment and decision-making tasks (Stewart et al, 2015), and this experience may influence the strength of different experimental manipulations (Chandler, Paolacci, Peer, Mueller & Ratliff, 2015;Rand et al, 2014). However, we believe that task experience would -if anything -reduce the differences we observed in the present studies.…”

Section: Limitationscontrasting

confidence: 52%

Ambiguity and expectation-neglect in dilemmas of interpersonal trust

Evans

Krueger

2017

Judgm. decis. mak.

View full text Add to dashboard Cite

Recent research suggests that people discount or neglect expectations of reciprocity in trust dilemmas. We examine the underlying processes and boundary conditions of this effect, finding that expectations have stronger effects on trust when they are made accessible and when they are provided as objective probabilities (Study 1). Objective expectations have stronger effects when they are based on precise, rather than ambiguous, probabilities (Study 2). We also find that trust decisions differ from individual risk-taking decisions: people are more willing to trust, and expectations have stronger effects on trusting behavior (Study 2). These results show that the availability and ambiguity of expectations shape trust decisions, and that people differentially weight expectations in dilemmas of trust and individual risk-taking.

show abstract

Section: Limitationscontrasting

confidence: 52%

Ambiguity and expectation-neglect in dilemmas of interpersonal trust

Evans

Krueger

2017

Judgm. decis. mak.

View full text Add to dashboard Cite

show abstract

“…Across seven studies conducted by Berinsky and colleagues (2012) with over 1,500 unique subjects, 30% of subjects had participated in more than one study (the mean number of studies completed per subject was 1.7). Similarly, Stewart et al (2015) find high rates of repeated participation within laboratories. Further, the majority of workers "follow" favorite requesters, and this practice is more common among the most prolific workers (Chandler et al, 2014).…”

Section: Introductionmentioning

confidence: 98%

“…In one study's sample, the average MTurk worker had completed a staggering 1,500 MTurk jobs, of which 300 were academic studies (Rand et al, 2014). Another recent analysis found that it takes about seven months for half of the pool of workers to be replaced (Stewart et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

“…The issue of prior exposure may be further exacerbated by the fact that the pool of available workers is smaller than might be assumed. A recent analysis using capturerecapture analysis found that the average lab samples from a pool of about 7,300 MTurk workers (so a lab's reach in practice is much smaller than the population of 50,000 advertised by Amazon; Stewart et al, 2015). Further, a small number of extremely active workers within this pool (sometimes referred to as "professional" subjects) are responsible for large proportion of study responses (Chandler et al, 2014;Berinsky, Huber & Lenz, 2012).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Investigating an alternate form of the cognitive reflection test

2016

View full text Add to dashboard Cite

Much research in cognitive psychology has focused on the tendency to conserve limited cognitive resources. The CRT is the predominant measure of such miserly information processing, and also predicts a number of frequently studied decision-making traits (such as belief bias and need for cognition). However, many subjects from common subject populations have already been exposed to the questions, which might add considerable noise to data. Moreover, the CRT has been shown to be confounded with numeracy. To increase the pool of available questions and to try to address numeracy confounds, we developed and tested the CRT-2. CRT-2 questions appear to rely less on numeracy than the original CRT but appear to measure closely related constructs in other respects. Crucially, substantially fewer subjects from Amazon’s Mechanical Turk have been previously exposed to CRT-2 questions. Though our primary purpose was investigating the CRT-2, we also found that belief bias questions appear suitable as an additional source of new items. Implications and remaining measurement challenges are discussed.

show abstract

“…Moreover, low-quality responses and suspicious accesses (e.g., duplicate IP addresses, server farms) are blocked. To mitigate the 'super-worker problem' (i.e., a small group of workers that completes a disproportionately high share of tasks) and increase the pool of available MTurk workers (e.g., Chmielewski & Kucker, 2020;Stewart et al, 2015), we blocked the top-2% of workers from participating in our study (who complete ~34% of the tasks; Robinson et al, 2019).…”

Section: Participantsmentioning

confidence: 99%

The fish can rot from the heart, not just the head: Exploring the detrimental impact of transgressions by leaders at multiple levels of an organization

Maskor

Fladerer

Fong

et al. 2022

British J Social Psychol

View full text Add to dashboard Cite

In this registered report, we examined the effect of transgressions committed by leaders working at different group levels within an organization on employee outcomes. Based on social identity theorizing, we argued that organizational leader transgressions would affect organizational members’ experiences only at the organizational level, but that workgroup leader transgressions would impact organizational members’ experiences at both workgroup and organizational levels. To test these ideas, we developed a 2 (leader group affiliation: workgroup vs. organizational) × 2 (leader behaviour: normative vs. transgressive) between‐subjects experimental paradigm. As hypothesized, both workgroup and organizational leader transgressions resulted in decreased organizational identification and perceived organizational leader effectiveness. Contrary to our prediction, transgressions of both workgroup and organizational leaders were similarly detrimental to workers’ workgroup identification. However, as predicted, a transgressive workgroup leader had a greater negative impact on perceived workgroup leader effectiveness than a transgressive organizational leader. When outliers were excluded, a workgroup leader’s transgression was found to be more detrimental to work performance than an organizational leader’s transgression. Overall, this study demonstrates that the transgressions of lower‐level workgroup leaders can be as detrimental – and in some cases more detrimental – to workers than the transgressions of higher‐level organizational leaders.

show abstract

The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

Cited by 214 publications

References 28 publications

Ambiguity and expectation-neglect in dilemmas of interpersonal trust

Ambiguity and expectation-neglect in dilemmas of interpersonal trust

Investigating an alternate form of the cognitive reflection test

The fish can rot from the heart, not just the head: Exploring the detrimental impact of transgressions by leaders at multiple levels of an organization

Contact Info

Product

Resources

About