The Generalizability of Survey Experiments

Abstract

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

Owning Immigration: Messenger Ethnicity, Issue Ownership, and Support for Latino Candidates

“…We find no reason to believe this might be the case regarding this sample. Authors like Krupnikov and Levine (2014) and Mullinix et al (2015) also find samples of convenience (like ours) resemble and are affected by information in similar ways as national representative samples. Additionally, a large majority of these students took part in this experiment in introductory American Government courses, which are required of all students at this university regardless of major.…”

How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies

“…There is some evidence that these demographic differences account for minimal differences in effect sizes between MTurk and other Internet survey platforms that claim representative samples (Weinberg et al 2014). Moreover, careful checks of moderating demographic variables that are not representative of the United States in MTurk samples and/or weighting may alleviate concerns regarding external validity (Mullinix et al 2016;Weinberg et al 2014). Second, concerns regarding internal validity may be exaggerated.…”

Judicialization and Its Effects: Experiments as a Way Forward

“…MTurk has rapidly grown in popularity due to its ease of use and relative affordability for researchers (up to thirty times less expensive than population-based samples in a recent comparison; see Mullinix et al 2015, p. 4). Nascent research suggests that the reliability of conclusions drawn from opt-in samples on MTurk are comparable to those from population-based sampling (Mullinix et al 2015). Public law scholars who attempt to understand the effects of judicialization should consider employing opt-in samples in survey experiments.…”