The Accuracy of Measurements with Probability and Nonprobability Survey Samples: Replication and Extension

MacInnis, Bo; Krosnick, Jon A.; Ho, Annabell Suh; Cho, Mu Jung

doi:10.1093/poq/nfy038

Cited by 108 publications

(78 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This finding is not in line with the majority of past findings suggesting that probability samples are more representative of national data than non-probability samples [e.g. [22,[67][68][69] but it is in line with a few other ones [e.g. [37].…”

Section: Discussioncontrasting

confidence: 89%

Telephone versus web panel National Survey for monitoring adoption of preventive behaviors to climate change in populations: a case study of Lyme disease in Québec, Canada

Domche

Valois

Canuel

et al. 2020

BMC Med Res Methodol

View full text Add to dashboard Cite

Background: To monitor the adoption of climate change adaptive behaviors in the population, public health authorities have to conduct national surveys, which can help them target vulnerable subpopulations. To ensure reliable estimates of the adoption of these preventive behaviors, many data collection methods are offered by polling firms. The aim of this study was to compare a telephone survey with a web survey on Lyme disease with regard to their representativeness. Methods: The data comes from a cross-sectional study conducted in the Province of Québec (Canada). In total, 1003 people completed the questionnaire by telephone and 956 filled in a web questionnaire. We compared the data obtained from both survey modes with the census data in regard to various demographic characteristics. We then compared the data from both samples in terms of self-reported Lyme disease preventive behaviors and other theoretically associated constructs. We also assessed the measurement invariance (equivalence) of the index of Lyme disease preventive behaviors across the telephone and web samples. Results: Findings showed that neither the telephone nor the web panel modes of data collection can be considered more representative of the target population. The results showed that the proportion of item nonresponses was significantly higher with the web questionnaire (5.6%) than with the telephone survey (1.3%), and that the magnitude of the differences between the two survey modes was nil for 19 out of the 30 items related to Lyme disease, and small for 11 of them. Results from invariance analyses confirmed the measurement invariance of an index of adaptation to Lyme disease, as well as the mean invariance across both samples.

show abstract

Section: Discussioncontrasting

confidence: 89%

Telephone versus web panel National Survey for monitoring adoption of preventive behaviors to climate change in populations: a case study of Lyme disease in Québec, Canada

Domche

Valois

Canuel

et al. 2020

BMC Med Res Methodol

View full text Add to dashboard Cite

show abstract

“…marital status, household income, number of adults living in the home, number of bedrooms in the home, employment status, smoking, drinking alcohol, possession of a passport, possession of a driver's license). Similar accuracy was observed recently by MacInnis et al (2018).…”

Section: Examined Surveys Conducted By Knowledgesupporting

confidence: 92%

“…This methodology is the bedrock of survey research that has yielded many of the most important findings in sociology, political science, economics, and other social science disciplines. Even today, despite response rates dropping progressively in recent decades (Keeter, Kennedy, Dimock, Best, & Craighill, 2006), true random sampling continues to yield strikingly accurate measurements of populations in surveys (MacInnis, Krosnick, Ho, & Cho, 2018;Yeager et al, 2011). 1 A few studies have explored the comparability of findings obtained from non-probability haphazard samples of adults with the findings obtained from college student participants (e.g., Peterson, 2001).…”

Section: Population: New Opportunities For Theoretical Advancementmentioning

confidence: 99%

“…As Paolacci and Chandler (2014) concluded, M-Turk study participants "should not be treated as representative of the general population" (p. 185), because they manifest considerable systematic biases similar to the characteristics of college student study participants: younger, more educated, under-representing African-Americans and Hispanics, more liberal, and more. Imposing quotas on demographics or weighting non-probability samples using demographics does not solve this problem (MacInnis et al, 2018;Yeager et al, 2011).…”

Section: More Use Of Representative Samplesmentioning

confidence: 99%

See 1 more Smart Citation

Moderation of classic social psychological effects by demographics in the U.S. adult population: New opportunities for theoretical advancement.

Yeager¹,

Krosnick²,

Visser³

et al. 2019

Journal of Personality and Social Psychology

View full text Add to dashboard Cite

For decades, social psychologists have collected data primarily from college undergraduates and, recently, from haphazard samples of adults. Yet researchers have routinely presumed that thusly observed treatment effects characterize "people" in general. Tests of seven highly-cited social psychological phenomena (two involving opinion change resulting from social influence and five involving the use of heuristics in social judgments) using data collected from randomly sampled, representative groups of American adults documented generalizability of the six phenomena that have been replicated previously with undergraduate samples. The one phenomenon (a cross-over interaction revealing an ease of retrieval effect) that has not been replicated successfully previously in undergraduate samples was also not observed here. However, the observed effect sizes were notably smaller on average than the meta-analytic effect sizes documented by past studies of college students. Furthermore, the phenomena were strongest among participants with the demographic characteristics of the college students who typically provided data for past published studies, even after correcting for publication bias in past studies using a new method, called the behaviorally-informed file-drawer adjustment (BIFDA). The six successful replications suggest that phenomena identified in traditional laboratory research also appear as expected in representative samples but more weakly, so observed effect sizes should be generalized with caution. The evidence of demographic moderators suggests interesting opportunities for future research to better understand the mechanisms of the effects and their limiting conditions.

show abstract

“…A number of studies have shown, however, that the application of corrections (e.g. quota-based recruitment and population weighting) to nonprobability samples is effective in producing prevalence estimates that match those estimated from probability samples (MacInnis, Krosnick, Ho, & Cho, 2018;Yeager et al, 2011). Additionally, the corrections applied in this study were specific to the U.S. population, and so results are unlikely to represent levels of awareness of the nicotine content of JUUL products in other countries.…”

Section: Discussionmentioning

confidence: 85%

Awareness of the presence of nicotine in the JUUL Brand of e-cigarette among adolescents, young adults, and older adults in the United States

McKeganey¹,

Russell²,

Haseen³

2020

Drugs: Education, Prevention and Policy

View full text Add to dashboard Cite

Nicotine warning labels on e-cigarette products can inform potential consumers that using e-cigarettes can cause addiction. This study examined awareness of the omnipresence of nicotine in the JUUL e-cigarette through a cross-sectional online survey of 4860 adolescents (13-17 years), 3746 young adults (18-24 years) and 5000 older adults (25-99 years) in the United States. Analyses also examined variation in nicotine awareness as a function of age, ownership, and past 30-day frequency of JUUL use. Results indicated fewer than half of JUUL ever-users and fewer than half of past 30-day JUUL users were aware that JUUL e-cigarettes always contain nicotine. Among JUUL ever-users, awareness of the omnipresence of nicotine in JUUL e-cigarettes was significantly lower among adolescents and those who did not personally own a JUUL. Among past 30-day JUUL users, awareness of the omnipresence of nicotine was significantly lower among adolescents and infrequent users (i.e. used on 1-19 of the past 30 days). Results suggest many users of a JUUL, especially youth, non-owners and infrequent users, may be underestimating their risk of becoming addicted to JUUL products, or overestimating their ability to stop using JUUL products whenever they choose. Reasons for low awareness and methods for increasing public awareness of the omnipresence of nicotine in JUUL products are proposed.

show abstract

The Accuracy of Measurements with Probability and Nonprobability Survey Samples: Replication and Extension

Cited by 108 publications

References 24 publications

Telephone versus web panel National Survey for monitoring adoption of preventive behaviors to climate change in populations: a case study of Lyme disease in Québec, Canada

Telephone versus web panel National Survey for monitoring adoption of preventive behaviors to climate change in populations: a case study of Lyme disease in Québec, Canada

Moderation of classic social psychological effects by demographics in the U.S. adult population: New opportunities for theoretical advancement.

Awareness of the presence of nicotine in the JUUL Brand of e-cigarette among adolescents, young adults, and older adults in the United States

Contact Info

Product

Resources

About