Demographics and Dynamics of Mechanical Turk Workers

Difallah, Djellel Eddine; Filatova, Elena; Ipeirotis, Panagiotis G.

doi:10.1145/3159652.3159661

Cited by 420 publications

(356 citation statements)

References 27 publications

Supporting

Mentioning

331

Contrasting

Unclassified

Order By: Relevance

“…Table provides basic demographic data for the 391 participants who completed both stages in our two experiments. The demographics of our sample are qualitatively similar to those of the mTurk population more broadly, as reported in Difallah et al ().…”

Section: Experimental Designsupporting

confidence: 80%

Financial Incentives and Herding: Evidence from Two Online Experiments

Bhanot

Williamson

2020

Southern Economic Journal

View full text Add to dashboard Cite

Social conformity and information‐based herding have been studied extensively in the social sciences, but there is little experimental evidence on how financial incentives impact the likelihood that an individual will follow the crowd. We present the results of a pair of two‐stage online experiments where we use answers to and confidence about trivia questions—with and without information about the choices of others—to test the impact of financial incentives on an individual's likelihood of engaging in herd behavior (i.e., switching their answer to the most popular answer when it is revealed). We find strong evidence that individuals are more likely to herd when there are financial incentives to be correct, suggesting that individuals are less likely to rely on their own beliefs and judgments when the stakes are higher. We also exploit the unique design of our experiments to show that in the absence of information about others' choices, men report higher levels of confidence than women.

show abstract

Section: Experimental Designsupporting

confidence: 80%

Financial Incentives and Herding: Evidence from Two Online Experiments

Bhanot

Williamson

2020

Southern Economic Journal

View full text Add to dashboard Cite

show abstract

“…Researchers generated an online platform called “MTurk‐tracker” that tracks and live releases demographic information of workers (eg, year of birth, gender, household size). Authors of this work demonstrated heterogeneity of workers in demographic propensities over time . Two studies report that the quality of the data collected from MTurk is comparable to that from other convenience samples such as undergraduate students or professional panels …”

Section: Methodsmentioning

confidence: 90%

“…Authors of this work demonstrated heterogeneity of workers in demographic propensities over time. 17 Two studies report that the quality of the data collected from MTurk is comparable to that from other convenience samples such as undergraduate students or professional panels. 17,18 We collected data from 702 respondents reached via MTurk.…”

Section: Survey and Cohortmentioning

confidence: 99%

“…17 Two studies report that the quality of the data collected from MTurk is comparable to that from other convenience samples such as undergraduate students or professional panels. 17,18 We collected data from 702 respondents reached via MTurk. Eligibility (adults over 18 and full-time residents of New Hampshire or Vermont) was assessed via screening and basic demographic questions.…”

Section: Survey and Cohortmentioning

confidence: 99%

See 1 more Smart Citation

Concordance of Rural‐Urban Self‐identity and ZIP Code‐Derived Rural‐Urban Commuting Area (RUCA) Designation

Onega

Weiss

Alford‐Teaster

et al. 2019

The Journal of Rural Health

View full text Add to dashboard Cite

Purpose This study examined the concordance between individuals’ self‐reported rural‐urban category of their community and ZIP Code‐derived Rural‐Urban Commuting Area (RUCA) category. Methods An Internet‐based survey, administered from August 2017 through November 2017, was used to collect participants’ sociodemographic characteristics, self‐reported ZIP Code of residence, and perception of which RUCA category best describes the community in which they live. We calculated weighted kappa (ĸ) coefficients (95% confidence interval [CI]) to test for concordance between participants’ ZIP Code‐derived RUCA category and their selection of RUCA descriptor. Descriptive frequency distributions of participants' demographics are presented. Findings A total of 622 survey participants, residents of New Hampshire (63%) and Vermont (37%), responded to the survey's self‐reported rural‐urban category. The overall ĸ was 0.33 (95% CI: 0.27‐0.38). The highest concordance was found among those living in a small rural area (N = 81, 13%): 62% of this group identified their communities as small rural. Sixty‐five percent (300/459) of participants residing in urban or large rural areas reported their community as more rural (small rural or isolated). Sixty‐eight percent (111/163) of participants living in small rural or isolated areas identified their community as more urban (large rural or urban). Conclusions Discordance was found between self‐report of rural‐urban category and ZIP Code‐derived RUCA designation. Caution is warranted when attributing rural‐urban designation to individuals based on geographic unit, since perceived rurality/urbanicity of their community that relates to health behaviors may not be reflected.

show abstract

“…The answer has important implications for research, but because Amazon does not provide such information obtaining an accurate answer is difficult. What is clear from published papers is that even though Amazon advertises more than 500,000 registered workers the number of active workers is considerably smaller [10,[21][22]. Previous studies using capture-recapture analysis have estimated, on the low end, that the average lab has access to less than 10,000 workers in any three-month span [10].…”

Section: Concerns About the Mturk Worker Poolmentioning

confidence: 99%

Tapped Out or Barely Tapped? Recommendations for How to Harness the Vast and Largely Unused Potential of the Mechanical Turk Participant Pool

Robinson¹,

Rosenzweig²,

Moss³

et al. 2019

Preprint

View full text Add to dashboard Cite

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool.Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

show abstract

Demographics and Dynamics of Mechanical Turk Workers

Cited by 420 publications

References 27 publications

Financial Incentives and Herding: Evidence from Two Online Experiments

Financial Incentives and Herding: Evidence from Two Online Experiments

Concordance of Rural‐Urban Self‐identity and ZIP Code‐Derived Rural‐Urban Commuting Area (RUCA) Designation

Tapped Out or Barely Tapped? Recommendations for How to Harness the Vast and Largely Unused Potential of the Mechanical Turk Participant Pool

Contact Info

Product

Resources

About