Responsive and Adaptive Survey Design: Use of Bias Propensity During Data Collection to Reduce Nonresponse Bias

Abstract

Reduction in nonresponse bias has been a key focus in responsive and adaptive survey designs, through multiple phases of data collection, each defined by a different protocol, and targeting interventions to a subset of sample elements. Key in this approach is the identification of nonrespondents who, if interviewed, can reduce nonresponse bias in survey estimates. From a design perspective, we need to identify an appropriate model to select targeted cases, in addition to an effective intervention (change in protocol). From an evaluation perspective, we need to compare estimates to a control condition that is often omitted from study designs, in addition to the need for benchmark estimates for key survey measures to provide estimates of nonresponse bias. We introduced a bias propensity approach for the selection of sample members to reduce nonresponse bias. Unlike a response propensity approach in which the objective is to maximize the prediction of nonresponse, this new approach deliberately excludes strong predictors of nonresponse that are uncorrelated with survey measures and uses covariates that are of substantive interest to the study. We also devised an analytic approach to simulate which sample members would have responded in a control condition. This study also provided a rare opportunity to estimate nonresponse bias, using rich sampling frame information, prior round survey data, and data from extensive nonresponse follow-up. The bias propensity model yielded reasonable fit despite the exclusion of the strongest predictors of nonresponse. The intervention was found to be effective in increasing participation among identified sample members. On average, the responsive and adaptive survey design reduced nonresponse bias by more than one-quarter—almost one percentage point—regardless of the choice of benchmark estimates. Effort under the control condition did not reduce nonresponse bias. While results are strongly encouraging, we argue for replication with varied populations and methods.

The Additional Effects of Adaptive Survey Design Beyond Post-Survey Adjustment: An Experimental Evaluation

“…Unfortunately, this experimental design can be in conflict with the practical goal of achieving the best survey outcomes. Survey organizations often opt for implementing RASD on the full sample to improve the recruitment, rather than holding out a control group to be recruited by the theoretically less efficient standard protocols (Axinn, Link, and Groves 2011;Bergmann and Scherpenzeel 2020;Peytchev et al 2020;Rosen et al 2014;van Berkel et al 2020). Without a control group, even though researchers may still be able to analytically tease out the effect of RASD on biases (Peytchev et al 2020;van Berkel et al 2020), it is no longer possible to study the effect of RASD together with post-survey adjustment.…”

“…First, the current adaptive design was implemented in a low-budget survey. This is an important difference from many other applications in large-scale well-funded surveys, which typically yield high response rates and leave limited room for improvement by RASD (Peytchev et al 2020). For example, a 2016 subsample of the National Household Education Survey experimented with an adaptive design on incentives; its total sample contained 206,000 addresses, and the final response rate was about 64% (Jackson et al 2020).…”

“…We combined the strategy of differential incentives with the strategy of tailoring invitation materials in mailings. The former is quite common (e.g., Han, Montaquila, and Brick 2013;Jackson et al 2020;Link and Burks 2013;Peytchev et al 2020;Singer et al 1999;Singer and Ye 2013), but the latter is a relatively novel technique (Lynn 2016).…”

“…Generally speaking, RASD refers to changing and tailoring recruitment procedures and protocols for different sample cases (Tourangeau et al 2017). One example strategy is differential incentiveshigher incentives are assigned to sample cases that are more reluctant to respond (Brick and Tourangeau 2017;Jackson, McPhee, and Lavrakas 2020;Link and Burks 2013;Peytchev, Pratt, and Duprey 2020;Singer, Groves, and Corning 1999;Singer and Ye 2013). To identify reluctant cases, survey practitioners rely on auxiliary information known for the entire sample and the outcomes of previous surveys (Tourangeau et al 2017).…”

Predicting Days to Respondent Contact in Cross-Sectional Surveys Using a Bayesian Approach

“…These parameters may be defined at various levels of the data collection process, including the interviewer level (Mneimneh et al 2018); the question-level (Mittereder and West 2021); the case-level (Wagner et al 2012), and the survey level (LaFlamme and Karaganis 2010). Indicators of cost, progress, and quality have also served as inputs into responsive and adaptive designs (Groves and Heeringa 2006;Schouten et al 2011) that allow for the balancing of cost and quality goals through centralized interventions including mode changes (Chesnut 2013;Coffey et al 2019), case prioritization (Walejko and Wagner 2018;Tolliver et al 2019;Peytchev et al 2020) and stopping rules (Wagner and Ragunathan 2010;Peytchev 2014).…”