A Bayesian Analysis of Design Parameters in Survey Data Collection

Abstract

In the design of surveys, a number of input parameters such as contact propensities, participation propensities, and costs per sample unit play a decisive role. In ongoing surveys, these survey design parameters are usually estimated from previous experience and updated gradually with new experience. In new surveys, these parameters are estimated from expert opinion and experience with similar surveys. Although survey institutes have fair expertise and experience, the postulation, estimation, and updating of survey design parameters is rarely done in a systematic way. This article presents a Bayesian framework to include and update prior knowledge and expert opinion about the parameters.

What Do You Think? Using Expert Opinion to Improve Predictions of Response Propensity Under a Bayesian Framework

“…In order to avoid anchoring bias while still eliciting reasonable responses, we provided an overall expected attempt-level response rate (24%), but did not provide anchor points for any particular category in the survey, allowing the experts to provide input for all items and categories. To avoid hindsight bias ( Schouten et al, 2018 ) arising from the fact that experts at ISR also conduct the NSFG, we recruited additional experts from the U.S. Census Bureau (Census). These additional experts have experience managing interviewer-administered data collections, but do not have experience with the NSFG or its data.…”

“…These additional experts have experience managing interviewer-administered data collections, but do not have experience with the NSFG or its data. By soliciting predictions from two geographically dispersed survey organizations with varying familiarity with the NSFG, we also hoped to protect against overconfidence bias ( Schouten et al, 2018 ), which can lead to prior distributions that are too narrow and do not accurately reflect the uncertainty in the prior.…”

“…In order to improve predictions, survey practitioners often use external data that may be more representative of a full data collection period. It is relatively common to estimate the coefficients of a predictive model using historical data, such as a prior implementation of the survey, and then apply those coefficients to the current round of data collection ( Schouten, Calinescu and Luiten 2013 ; Schouten, Wagner and Peytchev, 2017 ; Schouten, Mushkudiani, Shlomo, Durrant, Lundquist and Wagner, 2018 ). While this method provides data that might be representative of an entire data collection, it ignores current data in the prediction process.…”

“…Prior beliefs are generated from external data, most commonly historical data from the same survey, and those priors are then updated as the current data accumulates. Schouten et al (2018) discuss using Bayesian methods for predicting response and cost under different scenarios. Through simulation, they demonstrate value in the Bayesian methods in terms of reduced RMSE of predictions, while stressing that misspecification of the priors with respect to the true data should be relatively small.…”

“…Recently, survey researchers have introduced Bayesian approaches ( Schouten et al, 2018 ) to mitigate this bias by supplementing the current accumulating data with prior beliefs, generated from external data such as past implementations of the same survey or the survey methodological literature ( West, Wagner, Coffey and Elliott, 2019 ). While priors generated from past implementations of the same survey may be the most informative for a particular survey, that solution is not always an option.…”