Sample Representation and Substantive Outcomes Using Web With and Without Incentives Compared to Telephone in an Election Survey

Abstract

The objective of this article is to understand how the change of mode from telephone to web affects data quality in terms of sample representation and substantive variable bias. To this end, an experiment, consisting of a web survey with and without a prepaid incentive, was conducted alongside the telephone Swiss election survey. All three designs used identical questionnaires and probability samples drawn from a national register of individuals.First, our findings show that differences in completion rates mostly reflect different levels of coverage in the two modes. Second, incentives in the web survey strongly increase completion rates of all person groups, with the exception of people without Internet access or limited computer literacy. Third, we find voting behavior to be much closer to official figures in the web with the incentive version compared to the two other designs. However, this is partly due to the different sociodemographic compositions of the samples. Other substantive results suggest that the incentive version includes harder-to-reach respondents. Unit costs are much lower in the two web designs compared to the telephone, including when a relatively high incentive is used. We conclude that in countries with high Internet penetration rates such as Switzerland, web surveys are already likely to be highly competitive.

Who Gets Lost, and What Difference Does It Make? Mixed Modes, Nonresponse Follow-up Surveys and the Estimation of Turnout

“…the so-called "mobile generation" of people between 25 and 40 years and people with changing social relationships such as separation and divorce (Keeter 2007;Carley-Baxter, Peytchev, and Black 2010). Coverage problems also affect internet (web) surveys, which still underrepresent older people and overrepresent younger people, males, and highly educated persons (Stern and Dillman 2006;Zickuhr and Smith 2012;Lipps and Pekari 2016). Second, internet surveys tend to suffer from high nonresponse rates (Messer, Edwards, and Dillman 2012;Stern, Bilgen, and Dillman 2014;Roberts, Joye, and Ernst St€ ahli 2016), whereas telephone surveys have faced increasing refusal rates (de Leeuw 2008).…”

What Can Instrumental Variables Tell Us About Nonresponse In Household Surveys and Political Polls?

“…2003; Groves et al. 2006; Singer and Ye 2013; Lipps and Pekari 2016). These studies generally do not examine the way that induced changes in response rates alter the composition of the respondent sample.…”

“…One-third of the letters were randomly assigned to include a prepaid postal check for 20 CHF ($21.88 USD). The money was intended to encourage participation (Lipps and Pekari 2016). Every member of the sample is included in the data, which means that the SES data set includes information on incentive status and several register variables for both respondents and nonrespondents.…”

“…I apply the methods to data from a survey experiment conducted during the 2011 Swiss Electoral Study (SES) (Lipps and Pekari 2016). In the SES, prior to initial contact, members of the sample were randomly assigned to either a control group or a treatment group that received a small cash incentive to participate in the survey.…”

Advancing the psychology of social class with large-scale replications in four countries

“…Office drew a random sample (stratified by canton) of ≈ 50,000 Swiss addresses; (2) a public institution (the DAL) will print, fold, and assemble 50,000 letters of invitation to participate in our study; and (3) our university will send these 50,000 letters. We expect a response rate between 1 /5 and 1 /6 (the response rate for a Swiss probability-based web survey without monetary incentive 84 , if the final sample size is N ≥ 7,734 (response rate ≥ 15.45), the statistical power to detect a small-sized hypothesized effect on a given outcome will be .95, and we will stop data collection; however, if the final sample size is N < 7,734 (response rate < 15.45), the statistical power will be below .95, and we will ask Qualtrics to complete the sample (to reach N = 7,734) using the same quota sampling approach used in the U.S. and…”