Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics

Abstract

This article examines online recruitment via Facebook, Mechanical Turk (MTurk), and Qualtrics panels in India and the United States. It compares over 7300 respondents—1000 or more from each source and country—to nationally representative benchmarks in terms of demographics, political attitudes and knowledge, cooperation, and experimental replication. In the United States, MTurk offers the cheapest and fastest recruitment, Qualtrics is most demographically and politically representative, and Facebook facilitates targeted sampling. The India samples look much less like the population, though Facebook offers broad geographical coverage. We find online convenience samples often provide valid inferences into how partisanship moderates treatment effects. Yet they are typically unrepresentative on such political variables, which has implications for the external validity of sample average treatment effects.

The perceived impact of the 2016 election: A mediation model predicting health outcomes

“…Notably, our results are based on a sample of Amazon's Mechanical Turk (MTurk) respondents from the United States. While MTurk samples are ethnically, geographically, and socio-economically diverse, MTurk workers overall tend to be more politically knowledgeable (Berinsky et al, 2012), more liberal, younger, more educated, have lower income (Berinsky et al, 2012;Boas et al, 2018;Levay et al, 2016), and have higher levels of social anxiety (Shapiro et al, 2013) and depressive symptoms (Arditte et al, 2016) relative to nationally representative samples. The presence of slightly higher rates of participants reporting Democratic party identification supports the presence of a small liberal self-selection bias in this sample.…”

Why do cognitive prompts hurt learning in older adults?

“…The remaining participants were recruited through a Qualtrics study panel (https ://www.qualt rics.com/online-sampl e/), which are effective for recruiting hard to reach participants (Ibarra, Agas, Lee, Pan, & Buttenheim, 2018). Samples collected through Qualtrics have been shown to be demographically and politically representative as compared to national benchmarks (Baos, Christenson, & Glick, 2018). Qualtrics participants were paid for participation, although the company does not release information on exact payments.…”

Demography in the Digital Era: New Data Sources for Population Research

“…Finally, the Data Revolution has created new opportunities for collecting primary data using devices connected to the internet. Examples discussed in this paper include online surveys [7,1], apps for registering time-use data [28], and internet advertising platforms [37,9].…”

“…The Data Revolution has also created new opportunities for collecting primary data via the internet. Several studies have recruited participants for online surveys using social networking sites (e.g., FB and Twitter) and online labor markets (e.g., Amazon Mechanic Turk and Craigslist) [7]. Such platforms tend to have wide reach and often allow the targeting of individuals based on specific demographic characteristics, interests, and behaviors.…”