Cross-Sample Comparisons and External Validity

Abstract

Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

The Generalizability of Survey Experiments

“…While we could have compared the TESS sample directly to other convenience samples as we did in Study 1, we limited our focus to a single convenience sample (MTurk) in order to assess a larger number of issues in a manner that was feasible. MTurk is an increasingly popular avenue for experimental research across the social sciences (Bohannon 2011) and related research on the utility of the platform has been conducted but only with a small number of issues (Berinsky et al 2012;Krupnikov and Levine 2014). 7 We selected a total of 20 survey experiments that had been implemented using the TESS survey population sample platform.…”

“…MTurk is an online crowdsourcing platform that has become widely used across the social sciences due its ease of use, low cost, and capacity to generate more heterogeneous samples than subject pools of students (see Berinsky, Huber, and Lenz 2012;Krupnikov and Levine 2014;Paolacci, Chandler, and Ipeirotis 2010). That said, MTurk is an opt-in sample, meaning that respondents self-select into participating rather than being drawn with known probability from a well-specified population, and, as such, MTurk and other convenience samples invariably differ from representative population samples in myriad, possibly unmeasured, ways.…”

“…Berinsky, Huber, and Lenz 2012;Goodman, Cryder, and Cheema 2012;Horton, Rand, and Zeckhauser 2011;Krupnikov and Levine 2014;Paolacci, Chandler, and Ipeirotis 2010;Weinberg, Freese, and McElhattan 2014). 3 While these studies are impressive and telling, each includes only a small number of comparisons (e.g., three experiments) on a limited set of issues (e.g., three or four) and topics (e.g., question wording, framing) with few types of samples (e.g., three) at different points in time (e.g., data were collected on distinct samples far apart in time).…”

“…3 While these studies are impressive and telling, each includes only a small number of comparisons (e.g., three experiments) on a limited set of issues (e.g., three or four) and topics (e.g., question wording, framing) with few types of samples (e.g., three) at different points in time (e.g., data were collected on distinct samples far apart in time). Indeed, in one of the broader sample comparisons, Krupnikov and Levine (2014) conclude that their study with three samples (students, MTurk, and a population sample) is "only able to scratch the surface" (78).…”

Manipulated vs. Measured: Using an Experimental Benchmark to Investigate the Performance of Self-Reported Media Exposure

“…To date, scholars have either tried to "fix" the problems with self-reported exposure items (e.g., Bartels, 1993) or make theoreticallyinspired modifications to the wording of existing self-report measures, using convergent and/or predictive validity to gauge the degree of improvement (e.g., Althaus & Tewksbury, 2007;Dilliplane, Goldman, & Mutz, 2013;Prior, 2009a). 2 There is, however, a fundamental 1 Conceptually, benchmarking studies are related to a wave of recent research making crossstudy comparisons (e.g., Barabas & Jerit, 2010;Clifford & Jerit, 2014;Jerit, Barabas, & Clifford, 2013;Krupnikov & Levine, 2014). 2 Convergent validity refers to the fit between independent measures of the same construct, while predictive validity refers to the ability of a construct to predict some criterion variable (usually political knowledge in the case of media exposure; see Dilliplane, Goldman, & Mutz, 2013).…”