Comparing the Ability of Regression Modeling and Bayesian Additive Regression Trees to Predict Costs in a Responsive Survey Design Context

Abstract

Responsive survey designs rely upon incoming data from the field data collection to optimize cost and quality tradeoffs. In order to make these decisions in real-time, survey managers rely upon monitoring tools that generate proxy indicators for cost and quality. There is a developing literature on proxy indicators for the risk of nonresponse bias. However, there is very little research on proxy indicators for costs and almost none aimed at predicting costs under alternative design strategies. Predictions of survey costs and proxy error indicators can be used to optimize survey designs in real time. Using data from the National Survey of Family Growth, we evaluate alternative modeling strategies aimed at predicting survey costs (specifically, interviewer hours). The models include multilevel regression (with random interviewer effects) and Bayesian Additive Regression Trees (BART).

A Study of Interviewer Compliance in 2013 and 2014 Census Test Adaptive Designs

“…In only two of 16 separate experimental interventions were response rates significantly higher in the experimental group, which may have been due to lack of compliance. Interviewers made more calls on prioritized cases in all 16 experiments but in only seven were call attempts significantly higher (Wagner et al 2012).…”

“…First, we describe a case prioritization intervention. Other surveys have successfully implemented this type of intervention (Wagner et al 2012;Peytchev et al 2010). In this article, interviewers did not implement the intervention as designed.…”

Some Open Questions on Multiple-Source Extensions of Adaptive-Survey Design Concepts and Methods

“…Adaptive and responsive survey designs tailor data collection features to impact measures of data quality, such as nonresponse error or measurement error, or measures of cost (Schouten et al 2017). The tailoring decisions are anchored in specific pre-defined survey goals, such as increasing response rate or balance in the respondent population (Coffey et al 2020;Wagner et al 2012); reducing the variance of key survey estimates or the variation in weighting adjustments (Beaumont et al 2014;Paiva and Reiter 2017); or controlling specified data collection costs (Coffey and Elliott 2023;Peytchev 2014;Wagner et al 2023). As a result, adaptive and responsive designs typically increase effort (i.e., resources) to certain cases; and decrease effort in others.…”

“…Predicted or imputed responses, or information similar to key survey items, may be compared to accumulating response data to evaluate the quality of responses, and thus to inform intervention decisions (Morris et al 2015). Additionally, information about the costs of survey operations may be used to predict future data collection costs (Wagner 2019; Wagner et al 2020) to guide intervention decisions such as the optimal time to move cases from one less expensive phase of data collection to the next, potentially more expensive, phase (Wagner et al 2020).…”

“…To achieve both goals, the survey team developed prioritization rules that would help to redistribute resources to cases that were likely to respond and would increase representativeness (Tolliver et al 2017). Alternatively, Wagner et al (2012) attempted to maximize response rate, among other goals, by using response propensity to identify and prioritize the cases most likely to respond.…”