Responsive Survey Designs for Reducing Nonresponse Bias

Abstract

Survey researchers have been investigating alternative approaches to reduce data collection costs while mitigating the risk of nonresponse bias or to produce more accurate estimates within the same budget. Responsive or adaptive design has been suggested as one means for doing this. Falling survey response rates and the need to find effective ways of implementing responsive design has focused attention on the relationship between response rates and nonresponse bias. In our article, we re-examine the data compiled by Groves and Peytcheva (2008) in their influential article and show there is an important between-study component of variance in addition to the within-study variance highlighted in the original analysis. We also show that theory implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study. We then propose a typology of response propensity models that help explain the empirical findings, including the relative weak relationship between nonresponse rates and nonresponse bias. Using these results, we explore when responsive design tools such as switching modes, giving monetary incentives, and increasing the level of effort are likely to be effective. We conclude with some comments on the use of responsive design and weighting to control nonresponse bias.

Nonresponse Bias in Sample Surveys

“…If there is a sizeable difference between the means for respondents and nonrespondents, bias will be diminished as the response rate increases; however, it is quite possible to have both high nonresponse and low bias if the differences in means are small. In fact, several studies have found the correlation between response rates and nonresponse bias to be quite low, and often there is a great deal of variability in bias across different estimates in the same survey (Brick & Tourangeau, ; Fosnacht, Sarraf, Howe, & Peck, ; Groves & Peytcheva, ; National Research Council, ). If not careful, measures taken during data collection with a narrow focus on increasing response rates could actually increase bias (Groves, ).…”

“…It is critical that values for these variables are known for nearly all respondents and nonrespondents. It is important to also consider how the various targeted interventions may impact nonresponse among the key groups and how they will respond to the interventions (Brick & Tourangeau, ). As noted above, interventions focused solely on increasing response rates without taking into consideration the potential for nonresponse bias may be ineffective at reducing bias and could increase bias.…”

The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

“…For quite some time, paradata has been popular in the evaluation of data quality in a total survey error framework (for detailed overviews, see Kreuter & Olson, 2013;Olson & Parkhurst, 2013). Recently, researchers and survey organizations have started to shift the focus from post-survey analysis of paradata to an ongoing use of paradata during data collection, with the aim of monitoring and guiding the data collection process as the survey progresses (e.g., Brick & Tourangeau, 2017;Chun, Heeringa, & Schouten, 2018;Groves & Heeringa, 2006;Lepkowski et al, 2010;Schouten, Peytchev, & Wagner, 2017).…”

An Embedded Experiment for Targeted Non-Response Follow-Up in Establishment Surveys

“…Furthermore, even if the survey totals themselves appear to be robust to non-response thanks to the contribution from the largest units, consistently poor response in a specific subdomain such as small unit size strata is inherently tied to non-response bias, as discussed in Brick and Tourangeau (2017). Brick and Tourangeau (2017) re-examined the data that were used in the Groves and Peytcheva (2008) meta-analysis and drew a very different conclusion from the original analysis, developing theory that 'implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study '. Under this theoretical framework, if non-response is disproportionately high in a subdomain (e.g. small businesses in an industry stratum), then successful efforts that improve the subdomain non-response will have beneficial effects for the entire survey.…”