Accuracy in Estimation with Nonresponse: A Function of Degree of Imbalance and Degree of Explanation

Survey researchers have been investigating alternative approaches to reduce data collection costs while mitigating the risk of nonresponse bias or to produce more accurate estimates within the same budget. Responsive or adaptive design has been suggested as one means for doing this. Falling survey response rates and the need to find effective ways of implementing responsive design has focused attention on the relationship between response rates and nonresponse bias. In our article, we re-examine the data compiled by Groves and Peytcheva (2008) in their influential article and show there is an important between-study component of variance in addition to the within-study variance highlighted in the original analysis. We also show that theory implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study. We then propose a typology of response propensity models that help explain the empirical findings, including the relative weak relationship between nonresponse rates and nonresponse bias. Using these results, we explore when responsive design tools such as switching modes, giving monetary incentives, and increasing the level of effort are likely to be effective. We conclude with some comments on the use of responsive design and weighting to control nonresponse bias.

Section: Level Of Effortmentioning

confidence: 99%

Section: Level Of Effortmentioning

confidence: 99%

See 1 more Smart Citation

Responsive Survey Designs for Reducing Nonresponse Bias

Brick

Tourangeau

2017

“…The balance indicator and the distance indicator described in Särndal and Lundquist (2014) measure the degree to which the response set is similar to the full sample with respect to auxiliary variables or paradata available to all units on the frame. Balance B x ¼ x r 2 x s , the difference between mean value for respondents and mean value for sampled units where…”

Section: Indicators Involving the Response Indicator: Type (1) And Tymentioning

confidence: 99%

Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments

Thompson

Kaputa

2017

The U.S. Census Bureau is investigating adaptive Nonresponse Follow-Up (NRFU) strategies for single unit businesses in the 2017 Economic Census. These collection protocols require a suite of viable alternative procedures that can be implemented. With business surveys, the majority of cognitive research and nonresponse follow-up procedures focus on collection methods that obtain valid response data from the larger businesses, and there is relatively little quantitative or qualitative research for small businesses. Moreover, the contact methods for small businesses are often constrained by budget limitations. Business programs at the U.S. Census Bureau rely on mailed reminder letters and supplemental promotional materials, with options for certified and bulk mailings. To explore the benefits and disadvantages of the proposed alternative nonresponse follow-up procedures for small businesses, we conducted a field experiment embedded in the 2014 Annual Survey of Manufactures, an annual program that has similar data collection procedures and sampling units as the Economic Census. This article describes the study and presents the results, then discusses how the recommended nonresponse follow-up procedures are implemented in an adaptive collection design test presently being conducted in the 2015 Annual Survey of Manufactures.

“…Adaptive data collection can grant a degree of closeness of x r to x s , a relatively low IMB. One approach is the threshold method used for the empirical work in later sections, see Särndal and Lundquist (2014). To achieve perfect balance, x r ¼ x s , is unlikely in practice.…”

Section: The Response Imbalance Statisticmentioning

confidence: 99%

Inconsistent Regression and Nonresponse Bias: Exploring Their Relationship as a Function of Response Imbalance

Lundquist

2017

One objective of adaptive data collection is to secure a better balanced survey response. Methods exist for this purpose, including balancing with respect to selected auxiliary variables. Such variables are also used at the estimation stage for (calibrated) nonresponse weighting adjustment.Earlier research has shown that the use of auxiliary information at the estimation stage can reduce bias, perhaps considerably, but without eliminating it. The question is: would it have contributed further to bias reduction if, prior to estimation, that information had also been used in data collection, to secure a more balanced set of respondents? If the answer is yes, there is clear incentive, from the point of view of better accuracy in the estimates, to practice adaptive survey design, otherwise perhaps not.A key question is how the regression relationship between the survey variable and the auxiliary vector presents itself in the sample as opposed to the response. Strength in the relationship is helpful but is not the only consideration. The dilemma with nonresponse is one of inconsistent regression: a regression model appropriate for the sample often fails for the responding subset, because nonresponse is selective, non-random.In this article, we examine how nonresponse bias in survey estimates depends on regression inconsistency, both seen as functions of response imbalance. As a measure of bias we use the deviation of the calibration adjusted estimator from the unbiased estimate under full response. We study how the deviation and the regression inconsistency depend on the imbalance. We observe in empirical work that both can be reduced, to a degree, by efforts to reduce imbalance by an adaptive data collection.