Social Media, Web, and Panel Surveys: Using Non‐Probability Samples in Social and Policy Research

Abstract

The use of online surveys has grown rapidly in social science and policy research, surpassing more established methods. We argue that a better understanding is needed, especially of the strengths and weaknesses of non-probability online surveys, which can be conducted relatively quickly and cheaply. We describe two common approaches to non-probability online surveysriver and panel sampling-and theorize their inherent selection biases: namely, topical selfselection and economic self-selection. We conduct an empirical comparison of two river samples (Facebook and web-based sample) and one panel sample (from a major survey research company) with benchmark data grounded in a comprehensive population registry. The river samples diverge from the benchmark on demographic variables and yield much higher frequencies on non-demographic variables, even after demographic adjustments; we attribute this to topical self-selection. The panel sample is closer to the benchmark. When examining the characteristics of a non-demographic subpopulation, we detect no differences between the river and panel samples. We conclude that non-probability online surveys do not replace probability surveys, but augment the researcher's toolkit with new digital practices, such as exploratory studies of small and emerging non-demographic subpopulations.

Daily emotional well‐being during the COVID‐19 pandemic

“…While the survey company recruited the panel on a quota‐controlled basis to be representative of the adult population, our study sample was predominantly female indicating some selection effects. Hence, it is important to work with larger probability‐based samples to build up a full profile of everyday life during the crisis, and the current study should not be seen as a substitute for such work (Lehdonvirta, Oksanen, Räsänen, & Blank, 2020). However, we examine within‐person effects and did not identify evidence that the effects identified differed systematically between males and females.…”

Psychological distress among Italians during the 2019 coronavirus disease (COVID-19) quarantine

“…A web-based format was chosen over random-digit dialling for both cost considerations and time constraints. Research to date has shown that Internetmediated questionnaires have reliability and internal validity characteristics that are proportionate to traditional formats [44,45]. We adopted the snowball sampling strategy, with its strengths and weaknesses.…”

Crisis management and distrust: Study of an industrial accident in France

“…The publication, on October 1, 2019, of the list of products on site does not help reassure the public to such an extent that before the end of the first week following the event, the court of Rouen is seized of several complaints, a parliamentary information mission is created, the dissatisfaction of several impacted professions is expressed in various ways. Fire started A fire broke out on Thursday, September 26, 2019 at around 3:40 a.m. within the Lubrizol factory hitting the barrel crushing and storage facilities as well as a nearby warehouse belonging to the company SCMT (Shopping and transport company) 19 . The formal origin of the fire has not been established.…”

“…At the same time, the municipalities concerned, once informed, also initiate their PPI and respectively set up a crisis unit. For the commune of Le Petit-Quevilly, the chronology is as follows 22 : https://www.alternative76.fr/2019/09/26/incendie-de-l-usine-lubrizol-contexte-bilan-et-t%C3%A9moigages/ 18 https://fr.wikipedia.org/wiki/Chronologie de l%27incendie de l%27usine Lubrizol %C3%A0 Rouen and de ses cons%C3% A9quences, (French) 19 Fire at Lubrizol Rouen, France Population alert At 6:11 a.m., the city of Petit-Quevilly uses its SMS and voice call alert system to which residents who wish to subscribe (6,770 households 23 ). Some residents close to the factory got scared from 5:15 a.m. -5:30 a.m. At 7:10 a.m., the Seine-Maritime prefecture announces that it will set off the sirens at 7:45 a.m. to alert the population.…”

“…However, there is no universal definition of what constitutes an online survey. [19] uses this term to designate surveys where respondents are not only recruited via Internet but must also complete an online questionnaire. [20], for its part, include any survey method involving Internet during the dissemination, sampling or design of the survey.…”