Can Non-full-probability Internet Surveys Yield Useful Data? A Comparison with Full-probability Face-to-face Surveys in the Domain of Race and Social Inequality Attitudes

Abstract

The authors investigate the potential utility of Web-based surveys of non-full-probabilistically sampled respondents for social science research. Specifically, they compare demographic, attitude response, and multivariate model results produced by two distinct survey modalities: the traditional full-probability sample face-to-face survey and the non-full-probability Web survey. Using data from the 2009 Race Cues, Attitudes, and Punitiveness Survey (RCAPS), the 2008 General Social Survey (GSS), and the 2008 American National Election Study (ANES), the authors find that (1) the unweighted demographic differences between surveys tend to be slight; (2) in comparison with GSS and ANES respondents, RCAPS respondents are more interested in politics and ideologically polarized; (3) in comparison with ANES respondents, RCAPS respondents are more racially and socially conservative, often selecting the most extreme response option; (4) when the dependent variable is a more general and abstract measure of social attitudes, the multivariate models generated by the ANES and RCAPS show several differences that are trivial in magnitude, but when the dependent variable is a more specific and concrete measure, the models show remarkable similarity; and (5) RCAPS multivariate models consistently explain more variance than ANES models. Overall, these findings show both substantial similarities across the two survey modalities as well as a few clear, reasonably well-specified differences.

Public Support for Emergency Shelter Housing Interventions Concerning Stigmatized Populations

“…This was especially beneficial given that our study relies heavily on detailed vignettes. The findings of other studies also suggest that nonprobability Web samples have a high degree of reliability (Burhmester, Kwang, and Gosling, 2011), may yield better data quality than population-based Web samples (Weinberg, Freese, and McElhattan, 2014) and random digit dialing samples (Chang and Krosnick, 2009), and obtain similar experimental results to studies that use probability samples (Simmons and Bobo, 2015).…”

Toward a Bifurcated Theory of Emotional Deterrence

“…() tested the generalizability of MTurk samples and were able to replicate both the direction and the statistical significance of 29 (or 81 percent) of 36 treatment effects previously documented in probability samples of the American public (see also Weinberg, Freese, and McElhatten, ). More generally, across a variety of topics, researchers have found that the use of an online convenience sample most often allows for accurate inferences about the direction and approximate magnitude of relationships among variables (Ansolabehere and Schaffner, ; Bhutta, ; Pasek, ; Simmons and Bobo, ).…”

Race, prejudice and attitudes toward redistribution: A comparative experimental approach

“…While YouGov does not provide a true probability sample, research suggests that analyses of causal effects tend not to be influenced by potential selection biases (e.g., the tendency of online panelists to be more politically interested) (Simmons & Bobo ). Moreover, the YouGov matched samples have achieved impressive rates of predictive validity, accurately predicting the outcome of several national, statewide and local elections, with an average error rate comparable to what would be expected given random sampling (Rivers & Bailey ; Vavreck & Rivers ).…”